The DOI matches its Crossref record; 80 of the 105 references it cites were found in a registry or an index; none of the 15 sources its claims cite is checked for support yet.
ChatGPT: Jack of all trades, master of none
Open accessAbstract
OpenAI has released the Chat Generative Pre-trained Transformer (ChatGPT) and revolutionized the approach in artificial intelligence to human-model interaction. The first contact with the chatbot reveals its ability to provide detailed and precise answers in various areas. Several publications on ChatGPT evaluation test its effectiveness on well-known natural language processing (NLP) tasks. However, the existing studies are mostly non-automated and tested on a very limited scale. In this work, we examined ChatGPT’s capabilities on 25 diverse analytical NLP tasks, most of them subjective even to humans, such as sentiment analysis, emotion recognition, offensiveness, and stance detection. In contrast, the other tasks require more objective reasoning like word sense disambiguation, linguistic acceptability, and question answering. We also evaluated GPT-4 model on five selected subsets of NLP tasks. We automated ChatGPT and GPT-4 prompting process and analyzed more than 49k responses. Our comparison of its results with available State-of-the-Art (SOTA) solutions showed that the average loss in quality of the ChatGPT model was about 25% for zero-shot and few-shot evaluation. For GPT-4 model, a loss for semantic tasks is significantly lower than for ChatGPT. We showed that the more difficult the task (lower SOTA performance), the higher the ChatGPT loss. It especially refers to pragmatic NLP problems like emotion recognition. We also tested the ability to personalize ChatGPT responses for selected subjective tasks via Random Contextual Few-Shot Personalization, and we obtained significantly better user-based predictions. Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI. Our results provide the basis for a fundamental discussion of whether the high quality of recent predictive NLP models can indicate a tool’s usefulness to society and how the learning and validation procedures for such systems should be established.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper evaluates ChatGPT on 25 analytical NLP tasks and GPT-4 on five subsets, automating prompts and analyzing more than 49k responses. It finds ChatGPT never reaches SOTA performance, with an average loss of about 25%, and that losses are larger for harder, pragmatic, and especially emotion-related tasks.
Key findings
- ChatGPT underperformed SOTA on every task: average SOTA quality was 73.7% versus ChatGPT’s 56.5%, with an average loss of 25.5%; ChatGPT was also less stable (std. dev. 23.3% vs. 16.7%). Excluding eight emotion-related tasks, average loss fell to 12.8% (ChatGPT 69.7% vs. SOTA 80.0%).
- Loss correlated with task difficulty (Pearson 0.46); pragmatic tasks were harder on average (32.3% difficulty) than semantic tasks (15.6%), and emotion tasks had an average loss of 52.59% (ChatGPT 28.44% vs. SOTA 60.28%).
- Random Contextual Few-Shot Personalization improved user-based predictions: GoEmoPer3 was 4.08 percentage points better than GoEmoPer0, while AggressionPer and UnhealthyPer showed higher ChatGPT loss than their non-personalized counterparts by 3.25 and 12.55 percentage points, respectively.
- GPT-4 had small loss on semantic tasks (0.9%–6.4% for TextEntail and WNLI) but still high loss on emotional tasks (46.3%–56.3%); ChatGPT still outperformed GPT-4 on GoEmo and PolEmo. Qualitative analysis found bias, moralizing refusals, non-determinism, and cases where experts accepted ChatGPT answers over human labels (e.g., Unhealthy: 81% ChatGPT vs. 43% human).
Methods
- Tested 25 NLP datasets/tasks (23 English, 2 Polish), split into semantic and pragmatic categories, including sentiment, emotion, offensiveness, stance, WSD, NLI, and QA; some tasks used personalized user context.
- Automated prompting via an unofficial PyGPT API using up to 20 accounts, with manual post-processing; the paper reports over 38,000 prompts in the setup and more than 49k responses in the abstract.
- Compared ChatGPT to SOTA models by rerunning available source code or using reported metrics, using F1 Macro or accuracy and computing Loss = 100% × (SOTA − ChatGPT) / SOTA; GPT-4 was tested on five selected subsets.
Limitations
- Stated: prompts may not be strict or precise enough; post-processing was sometimes required; ChatGPT produced unexpected out-of-range responses; internal policy and biases affected answers; context was generally respected but offered limited control; efficiency and costs were constrained by lack of an official API, with about 2–3k prompts per day achieved.
- Stated: the difficulty analysis is limited because only 25 tasks were considered; GPT-4 testing was restricted by access limits (25 prompts per 3 hours).
- Evident from the text: dataset availability for ChatGPT training was assessed manually and is uncertain; qualitative expert evaluation covered 100 randomly selected cases per dataset and involved subjective judgments; ChatGPT showed non-determinism, with results varying across launches.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
98 Own result · 37 Own interpretation · 34 No basis given · 11 Own method · 10 Not checked yet
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For GPT-4 model, a loss for semantic tasks is significantly lower than for ChatGPT.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We showed that the more difficult the task (lower SOTA performance), the higher the ChatGPT loss.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Simultaneously, ChatGPT was less stable: the standard deviation of its performance was 23.3% compared to only 16.7% for the SOTA solutions.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“because for all of these datasets, we observe a significant drop in quality relative to state-of-the-art models.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It especially refers to pragmatic NLP problems like emotion recognition.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also tested the ability to personalize ChatGPT responses for selected subjective tasks via Random Contextual Few-Shot Personalization, and we obtained significantly better user-based predictions.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It is usually greater for semantic tasks rather than for pragmatic ones, which is related to the task difficulty, see Sec. 6.2.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our comparison of its results with available State-of-the-Art (SOTA) solutions showed that the average loss in quality of the ChatGPT model was about 25% for zero-shot and few-shot evaluation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“yet it has them in topics regarding ideological views.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 46
- Probably not a substantial claim. 70
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The crucial finding from our studies is that the ChatGPT performance is always lower than the SOTAmethods (loss>0) in all the tasks considered.”
- Also stated in: § 6.1. Jack of all trades, master of none · p. 10 · plain
“It means that ChatGPT never reached the level of the best existing models.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 6.1. Jack of all trades, master of none · p. 10 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The loss for most tasks did not exceed 25%. It was greater only for three problems: GoEmotions, PolEmo, and TweetEmoji.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The average quality of SOTA methods was at 73.7%, whereas ChatGPT was at only 56.5%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while GPT-4 achieved significantly higher scores in the remaining three semantic tasks with even a 9.9 p.p. increase for the WNLI task.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While solving mathematical calculations (SQuAD), the highest loss was among semantic tasks: 23.7%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, pragmatic tasks are more difficult (average difficulty = 32.3%), while the average difficulty for semantic tasks is only 15.6%.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also noticed that when it is unable to perform a task on the provided example, it will refuse to do so and provide an explanation why”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Those situations are usually solved by supplementing the model with information.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It comes especially from the emotional tasks, which are pragmatic and very difficult (average 39.7%).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the case of mathematical reasoning, it can provide a whole explanation of how it reached a certain outcome and provide only the answer without explanation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When exploiting the possibilities of ChatGPT, we can see that it can perform various tasks, including recognizing generalized and personalized dimensions of Natural Language Processing, answering questions where a generous amount of domain knowledge is required, or even writing lines of code in the programming language of choice.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can also observe that the loss is correlated with the task difficulty; see Fig. 7. The Pearson correlation coefficient between difficulty and loss is equal to 0.46.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When conducting experiments regarding tasks such as humor recognition or offensiveness detection, we have stumbled upon output that not only refuses to answer whether something is or is not funny but also sends a moralizing message with an irritated tone.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ChatGPT can also return a number as a category indicator or whole output in the JSON format.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In such a case, the average loss is reduced by as much as half, from 25.5% to 12.8%; the difference in performance drops from 17.2pp to 10.3pp.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If we discard all eight emotion-related tasks (ids: 12-17, 20-21), the average SOTA performance reaches 80%(increase by 6.3pp), but ChatGPT improves much more: by 13.2pp, up to 69.7%.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ChatGPT will only generate the content if the user adequately motivates it with the scientific goals.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, in the case of the ChatGPT model, prompt tuning is technically unavailable”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“as it has happened in the case of ClarinEmo B.12, where the model stated that all provided texts are legal and financial statements.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Anyway, the differences in performance are not significant.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Onthe other hand, ChatGPT demonstrated its instability and tendency towards non-determinism.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It also refers to the necessary manual correction of typos surprisingly provided by ChatGPT”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, prompts themselves and the sequence of prompts provide limited control over them.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“So it evaluates the main content but ignores the broader context (see Chat 64).”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Therefore, some responses are locked due to the OpenAI policy.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“e.g., the unlisted emotion satisfaction has been delivered:”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In this example, the sentence was interpreted as not aligned with society's standards.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“## Who is right? (expert rate)”
- Also stated in: § Expected · p. 41 · plain
“Evaluation result human - not correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Also stated in: § Expected · p. 41 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“## Who is right? (expert rate)”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluation result human - not correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“## Who is right? (expert rate)”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“human - correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluation result human - not correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The annotator has evaluated the below message as expressing gratitude, whereas ChatGPT regards it as neutral (Chat 58)”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“3. ChatGPT flattens the message, partially ignoring the metatext.”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, ChatGPT explicitly distances itself from judging people.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ChatGPT labeled as aggressive only 11 texts from the WikiDetox Aggression dataset labeled by people as non-aggressive, while the opposite decision was taken 207 times.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluation result human - not correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Restrictive or wrongly implemented policy will influence the user experience of the ChatGPT by not answering all user's prompts exaggeratedly judging many questions not along with OpenAI rules:”
hedgeddirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“human - correct, ChatGP - not correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“when asked which language phenomena demonstrate the enthusiasm or positive sentiment in the text, it gives a list containing such phenomena as the use of superlatives or lack of negative language”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Context is generally respected by ChatGPT.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“human - correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The problem of controversial and ambiguous utterances is solved by ChatGPT's demonstration of a lower level of confidence”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“## Who is right? (expert rate)”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“human - not correct , ChatGPT - not correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“## Who is right? (expert rate)”
plainSingle case- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As mentioned in the previous section, ChatGPT tends to find negative connotations in the given text.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63:”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Similarly, ChatGPT justifies the negative sentiment of the news about the ban on naming streets after Fidel Castro, Chat 66:”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Internal policy can strongly influence the answers received.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All these tasks are related to a very subjective problem of emotional perception and individual interpretation of the content.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“In cases where the performance of ChatGPT was compared to other solutions (e.g. [26, 30, 41]), it worked on a comparable level to the competitor but not outperforming any major SOTA solutions.”
What the sources say
- Not checked yetcites [26] Kung, 2022
- Not checked yetcites [30] Goyal, 2022
- Not checked yetcites [41] Jiao, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“There were a couple of large-scale evaluations published recently [46, 47] and both of them indicate that while the performance of ChatGPT is significant it doesn't outperform the SOTA solutions with the exception of sentiment analysis task in case of [46] which suggests it is a significant area for further research.”
What the sources say
- Not checked yetcites [46] Amin, 2023
- Not checked yetcites [47] Bang, 2023
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
In the paper
“ChatGPT's successor, the GPT-4 [19], is most likely an even larger model that can additionally receive not only text but also images as input.”
What the sources say
- Not checked yetcites [19] OpenAI, 2023
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Language models are prone to generating responses containing human-like biases as well as presenting moral and ethical stances [23].”
What the sources say
- Not checked yetcites [23] Schramowski, 2022, Large pre-trained language models contain human-like biases of what is right and wrong to do, Nat. Mach. Intell. source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even though the creators of ChatGPT secure the model against answers that are offensive, there are still multiple methods that may reveal its hidden biases.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nevertheless, we can state that ChatGPT performs pretty well on all tasks except emotional ones.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, the number of cases that required post-processing was relatively small (column #Post-processed in Tab. 1). For most tasks (16), the contribution of such texts was less than 5%. Only for Aggression, SQuAD, Unhealthy, and ReAding, it exceeded 15%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Manual post-processing correctly extracted and mapped ChatGPT's raw outputs to the intended labels.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Simultaneously, its achievements are always below SOTA but usually not so much.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such results prove that ChatGPT is Jack of all trades, master of none .”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, using the prompts directly as humans designed them implicitly allows us to evaluate models' language comprehension abilities.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We emphasize that the results for GPT-4, as for ChatGPT, could significantly differ, if distinct prompt schemes are compared, see Sec. 6.6.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“If possible, we launched our models equivalent to SOTA solutions since the setup (especially data split) was often different than in the original paper. For that purpose, we usually utilized source codes published by the authors. Unfortunately, it was impossible for some tasks, so we exploited the performance results provided in the original paper.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: The SOTA results taken from original papers are comparable to the paper's own setup despite differences in data split and evaluation.
In the paper
“Borji [105] conducted a systematic review of the typical categories of ChatGPT failures.”
What the sources say
- Not checked yetcites [105] Korczyński, 2022, Compression methods for transformers in multidomain sentiment analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The above errors are derived from both incorrect reasoning in terms of psychological reasoning and bias and discrimination.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This analysis, however, requires further investigations since the number of the tasks considered (25) still remains relatively small.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is observable that semantic tasks (blue crosses) are rather easy; hence, their ChatGPT loss is relatively small.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The annotations of six independent annotators were aggregated to label each sentence with all potential options, using the label when at least two annotators agreed on it.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using the glosses to explain senses to a language model implicitly tests its language comprehension abilities.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“It is our new dataset that has not yet been published. We exploited this dataset to ensure that ChatGPT was not trained on it.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: ChatGPT was not trained on the ClarinEmo dataset.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and the only way to verify prompt relevance is to evaluate its performance directly in the downstream task.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“If available, we tried to validate ChatGPT using one measure F1 Macro, which is commonly acceptable for imbalanced data, Tab. 2.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In the case of CoLa, WNLI, WordContext, and MathQA, we had to rely on the accuracy, as it was the only one presented in the reference paper; we could not replicate their studies and calculate our measures. WNLI and WordContext have their two classes balanced, so it is not an issue.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“which further implies possible consequences regarding clashes with fake news.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It may suggest that providing some answerable questions helps it detect unanswerable ones with the same context.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Demonstration-based personalization included in our prompts can be treated as similar to few-shot learning, even an extended context to a given prompt.”
plaincorrelationaln = 300- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Hagendorff [104] drew attention to the fact that chatbot ethics can be a subject of debate in fairness, nondiscrimination, and justice.”
What the sources say
- Not checked yetcites [104] Borji, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, the raw text provided by ChatGPT may require extraction of crucial information, e.g., final labels.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Information about the distinction for a particular footballer is neutral.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“1. Prompts may not be strict and precise enough.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“2. ChatGPT seems to have no regard for individuals, instead judging situations.”
hedgedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, this often leads to mistakes”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“6. ChatGPT presents the sense of common human morality.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ChatGPTrequires prompts in natural language, which is - by nature - not structured and can be different for different users and tasks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“A small difference in these two values proves a similar distribution of classes in both sets, thus, a good stratification of sampling.”
- Also stated in: § Table 8 · p. 28 · boosted
“A marginal difference in entropy of output real values for test or dev set (column #Test in Tab. 1) and within the set used by us (column #Used in Tab. 1) demonstrates a good stratification of the used set selection.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Table 8 · p. 28 · boosted
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It also depends on the prompting quality and correctness.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its sentiment, however, can be both positive and negative. It depends on the sympathies of the recipient sympathies regarding specific footballers.”
plainconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT evaluates rather situations than participants, but words refer to people, which lead to a specific, predetermined assessment”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Analysis was conducted by experts who are specialists trained in the recognition of emotions in the text. One of them is a psychologist and another is a linguist, both are experienced annotators.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It also provides a unique selfexplanation capability that facilitates human understanding and adaptation to the expected outcome.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, we are dealing with self-explanatory artificial intelligence, which is a part of eXplainable Artificial Intelligence (XAI)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It includes various biases and political correctness incorporated into the general ChatGPT model.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper relies on this without support.
The 25 selected NLP tasks are representative of typical NLP analytical tasks.
“We tested ChatGPT on 25 tasks focusing on solving common NLP problems and requiring analytical reasoning, Tab. 1.”
The paper relies on this without support.
ChatGPT's self-reports about whether it used a dataset for training are reliable.
“Additionally, we asked ChatGPT whether or not the dataset was used for training.”
The paper relies on this without support.
ChatGPT was not trained on the ClarinEmo dataset.
“It is our new dataset that has not yet been published. We exploited this dataset to ensure that ChatGPT was not trained on it.”
The paper relies on this without support.
The unofficial PyGPT API and the use of up to 20 accounts produced outputs equivalent to the official ChatGPT service.
“Without an official API, we modified and used an unofficial API called PyGPT 5 , written in Python. During the research, we exploited up to 20 accounts to gather data regarding 25 datasets.”
The paper relies on this without support.
The SOTA results taken from original papers are comparable to the paper's own setup despite differences in data split and evaluation.
“If possible, we launched our models equivalent to SOTA solutions since the setup (especially data split) was often different than in the original paper. For that purpose, we usually utilized source codes published by the authors. Unfortunately, it was impossible for some tasks, so we exploited the performance results provided in the original paper.”
The paper relies on this without support.
Manual post-processing correctly extracted and mapped ChatGPT's raw outputs to the intended labels.
“This imposes a necessity to check what happened and extract answers from ChatGPT output manually. The next step is to cast the resulting outputs to the correct labels in the dataset.”
The paper relies on this without support.
The performance of modern language models such as T5, GPT-3, and ChatGPT heavily relies on the quality of task-specific prompts.
“The performance of modern language models, such as T5, GPT-3, and ChatGPT, heavily relies on the quality of task-specific prompts.”
The paper relies on this without support.
The prompt-based learning paradigm requires careful prompt engineering and prompt tuning.
“The prompt-based learning paradigm requires careful prompt engineering and prompt tuning.”
The paper relies on this without support.
Most tasks require a prompt that enables the model to choose a certain value from the provided options.
“Most tasks require a prompt that enables the model to choose a certain value from the provided options.”
The paper relies on this without support.
The annotations in the selected 4 datasets were of a more or less subjective nature, and detailed guidelines were not necessary to achieve high interannotator agreement.
“The annotations in the selected 4 datasets were of a more or less subjective nature, and for this reason, it was not necessary to create detailed guidelines in order to achieve high interannotator agreement”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In a large study of ChatGPT that included 25 tasks and over 48,000 prompts, the tool was found to perform reasonably well when conducting quantitative analysis, with a somewhat poorer performance with tasks that require more reasoning ( Kocoń et al., 2023).”
From Generative AI for corpus approaches to discourse studies: A critical evaluation of ChatGPT (2024) · cites this paper as Kocoń et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This underperformance by the chat model is consistent with the findings by Kocon et al. [39].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [39]
In the citing paper
“Similarly, LLMs’ performance in logic and deductive tasks is often poor (64, 65), so using them for analysis may lead to false conclusions.”
From How should the advancement of large language models affect the practice of science? (2025) · cites this paper as [64]
In the citing paper
“Kocoń et al. (76) found that LLMs may struggle with complex, subjective tasks such as emotion recognition, and Huang et al. (18) found that LLMs can reinforce lay perspectives when detecting implicit hate speech and may thus mislead nonexperts with their explanations in cases where they make incorrect decisions.”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [76]
In the citing paper
“However, this perspective of LLMs, as a convenient and comprehensive tool in psychological text analysis, is challenged by recent critiques emphasizing their limitations, such as inconsistencies in text annotations, difficulties in explaining complex constructs such as implicit hate speech, and a potential lack of depth in specialized or sensitive areas (18, 76, 77).”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [76]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| 2 | AggressionPer | Pragmatic | F1 Macro | Our | 72.57 | 81.03 | 8.46 | 19.97 | 10.44 |”
Reported 19.97 · recomputed 18.97
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 6.3. Random Contextual Few-Shot Personalization · p. 12
“| 24 | ReAding | Semantic | F1 Macro | Our | 76.36 | 84.71 | 8.35 | 15.29 | 9.86 |”
§ Prompt · p. 29
“| 24 | ReAding | 76.36 | 76.34 | - | - | 84.71 | - |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 6.3. Random Contextual Few-Shot Personalization · p. 12
“| 8 | TextEntail | Semantic | F1 Macro | Paper | 88.09 | 92.10 | 4.01 | 7.90 | 4.35 |”
§ Prompt · p. 29
“| 8 | TextEntail | 88.09 | 87.88 | - | - | 92.1 | - |”
Citations
Cites
1 retracted
RETRACTED: WELMSD – word embedding and language model based sarcasm detection
Kumar, P., & Sarin, G. (2022) · Online Information Review · doi:10.1108/oir-03-2021-0184 · Cited by 17 · cited 1×
“3%) 995 2 20137 /19643 5 Sarcasm P EN Humor recognition No Binary classification Sarcasmania [67] / [68] 3 Yes 5967 1000 10 61 (6.”
BackgroundRetractedBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kilgarriff, A. (2000) · Language · doi:10.2307/417141 · Cited by 11,529
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
A survey of deep neural network architectures and their applications
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.038 · Cited by 3,239
Research works printed but not linked: 6
- Johnson, 2016, Supervised and semi-supervised text categorization using LSTM for region embeddings, vol. 48, 526
- Gillioz, 2020, Overview of the transformer-based models for NLP tasks, vol. 21, 179
- Radford, 2018, Improving language understanding by generative pre-training, OpenAI
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI
- Ngo, 2022, StudEmo: A non-aggregated review dataset for personalized emotion recognition, 46
- Fellbaum, 1998, Wordnet: An electronic lexical database, Comput. Linguist., 292
Cited by
Sorted by citations
Items 101–200 of 724
ChatGPT’s applications in marketing: a topic modeling approach
Tafesse, W., & Wien, A. H. (2024) · Marketing Intelligence & Planning · doi:10.1108/mip-10-2023-0526 · Cited by 36
Towards a new paradigm in intelligence-driven computational fluid dynamics simulations
Chen, X., Wang, Z., Deng, L., Yan, J., Gong, C., Yang, B., Wang, Q., Zhang, Q., Yang, L., Pang, Y., et al. (2024) · Engineering Applications of Computational Fluid Mechanics · doi:10.1080/19942060.2024.2407005 · Cited by 36
“…text generation (W. Yu, Zhu, et al., 2022), machine translation (Stahlberg, 2020), sentiment analysis (Taboada, 2016), and question-answering systems (Kocon et al., 2023), leading to a flourishing period for Artificial Intelligence Generated Content (AIGC) in both academic and industrial domains.”
Shan, S., & Li, Y. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2423335 · Cited by 36
“Compared to humans, it also exhibits clear advantages in response speed and breadth of knowledge (Koco � n et al., 2023).”
An Overview of Deep Neural Networks for Few-Shot Learning
Zhao, J., Kong, L., & Lv, J. (2024) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020049 · Cited by 33
ChatGPT, the perfect virtual teaching assistant? Ideological bias in learner-chatbot interactions
Poucke, M. V. (2024) · Computers & composition/Computers and composition · doi:10.1016/j.compcom.2024.102871 · Cited by 46
“Keep your friends close, but your enemies closer:” ChatGPT in tourism and hospitality
Altınay, Z., Altınay, F., Tlili, A., & Vatankhah, S. (2024) · Journal of Hospitality and Tourism Technology · doi:10.1108/jhtt-03-2024-0139 · Cited by 36
El-Akhras, H. A., El-Wahab, M. A. A., Saghier, E. G., & Selem, K. M. (2024) · Journal of Hospitality and Tourism Insights · doi:10.1108/jhti-04-2024-0292 · Cited by 33
Artificial intelligence and ChatGPT are fostering knowledge sharing, ethics, academia and libraries
Zeb, A., Rehman, F. U., Othayman, M. B., & Rabnawaz, M. (2024) · International Journal of Information and Learning Technology · doi:10.1108/ijilt-03-2024-0046 · Cited by 44
Alomari, E. A. (2024) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2024.052256 · Cited by 37
Cao, M., Wang, Q., Zhang, X., Liang, Z., Qiu, J., Yung, P. S., & Ong, M. T. (2024) · Journal of sport and health science/Journal of Sport and Health Science · doi:10.1016/j.jshs.2024.101016 · Cited by 32
Evaluating large language models on geospatial tasks: a multiple geospatial task benchmarking study
Xu, L.-C., Zhao, S., Lin, Q.-M., Chen, L., Luo, Q.-Q., Wu, S.-S., Ye, X., Feng, H.-L., & Du, Z.-H. (2025) · International Journal of Digital Earth · doi:10.1080/17538947.2025.2480268 · Cited by 40
ProLLaMA: A Protein Large Language Model for Multitask Protein Language Processing
Lv, L., Lin, Z., Li, H., Liu, Y., Cui, J., Chen, C. Y., Yuan, L., & Tian, Y. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3564914 · Cited by 93
Voices in the digital storm: Unraveling online polarization with ChatGPT
Xing, Y., Zhang, Z., Teng, G., & Zhou, X. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102534 · Cited by 28
Foppiano, L., Lambard, G., Amagasa, T., & Ishii, M. (2024) · Science and Technology of Advanced Materials Methods · doi:10.1080/27660400.2024.2356506 · Cited by 47
Wang, L., Dong, S., Chen, Y., Meng, X., Fang, S., & Fei, S. (2024) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2024.3477548 · Cited by 32
Personalized Large Language Models
Woźniak, S., Koptyra, B., Janz, A., Kazienko, P., & Kocoń, J. (2024) · 2024 IEEE International Conference on Data Mining Workshops (ICDMW) · doi:10.1109/icdmw65004.2024.00071 · Cited by 28
Self-citationGbenga-Ilori, A. O., Imoize, A. L., Noor, K., & Adebolu-Ololade, P. O. (2025) · AI · doi:10.3390/ai6060126 · Cited by 28
ChatGPT: Its Applications and Limitations
Chowdhury, M. N.-U.-R., & Haque, A. (2023) · International Conference on Intelligent Technologies (CONIT) · doi:10.1109/conit59222.2023.10205621 · Cited by 32
Anas, M., Saiyeda, A., Sohail, S. S., Cambria, E., & Hussain, A. (2024) · IEEE Intelligent Systems · doi:10.1109/mis.2024.3374582 · Cited by 31
“Personhood and AI: Why large language models don’t understand us”
Browning, J. (2024) · AI & SOCIETY · doi:10.1007/s00146-023-01724-y · Cited by 27
Commonsense-based adversarial learning framework for zero-shot stance detection
Zhang, H., Li, Y., Zhu, T., et al. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126943 · Cited by 26
Annamalai, N. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2024.2371928 · Cited by 34
Andersson, M., & McIntyre, D. (2025) · Journal of Pragmatics · doi:10.1016/j.pragma.2025.02.001 · Cited by 37
Evading text based emotion detection mechanism via adversarial attacks
Bajaj, A., & Vishwakarma, D. K. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126787 · Cited by 26
Public attitudes toward chatgpt on twitter: sentiments, topics, and occupations
Koonchanok, R., Pan, Y., & Jang, H. (2024) · Social Network Analysis and Mining · doi:10.1007/s13278-024-01260-7 · Cited by 25
Xu, Y., Liu, Q., Pang, J., Zeng, C., Ma, X., Li, P., Ma, L., Huang, J., & Xie, H. (2024) · Journal of Multidisciplinary Healthcare · doi:10.2147/jmdh.s477452 · Cited by 31
Estimating the quality of published medical research with ChatGPT
Thelwall, M. A., Jiang, X., & Bath, P. A. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104123 · Cited by 32
Can ChatGPT’s Responses Boost Traditional Natural Language Processing?
Amin, M. M., Cambria, E., & Schuller, B. W. (2023) · IEEE Intelligent Systems · doi:10.1109/mis.2023.3305861 · Cited by 30
A New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection
Yang, S., Sun, R., & Wan, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.256 · Cited by 46
Performance of ChatGPT on Chinese master’s degree entrance examination in clinical medicine
Li, K.-C., Bu, Z.-J., Shahjalal, M., He, B.-X., Zhuang, Z.-F., Li, C., Liu, J.-P., Wang, B., & Liu, Z.-L. (2024) · PLOS ONE · doi:10.1371/journal.pone.0301702 · Cited by 23
Atiyah, A. G., Faris, N. N., Rexhepi, G., & Qasim, A. J. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-51716-7_9 · Cited by 26
ChatGPT vs human expertise in the context of IT recruitment
Szandała, T. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125868 · Cited by 29
Workplace security and privacy implications in the GenAI age: A survey
Diro, A. A., Kaisar, S., Saini, A., Fatima, S., Pham, H., & Erba, F. (2025) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2024.103960 · Cited by 31
Buldur, M., & Sezer, B. (2024) · BMC Oral Health · doi:10.1186/s12903-024-04358-8 · Cited by 28
Take It, Leave It, or Fix It: Measuring Productivity and Trust in Human-AI Collaboration
Qian, C., & Wexler, J. (2024) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3640543.3645198 · Cited by 39
Xing, J., & Jiang, Q. (2024) · Kybernetes · doi:10.1108/k-10-2023-2237 · Cited by 28
High School Teachers’ Adoption of Generative AI
Elstad, E., & Eriksen, H. (2024) · Nordic Journal of Comparative and International Education (NJCIE) · doi:10.7577/njcie.5736 · Cited by 28
Shamim, M. S., Zaidi, S. J. A., & Rehman, A. (2024) · Journal of College of Physicians And Surgeons Pakistan · doi:10.29271/jcpsp.2024.05.595 · Cited by 27
RoBERTa, ResNeXt and BiLSTM with self-attention: The ultimate trio for customer sentiment analysis
Lak, A. J., Boostani, R., Alenizi, F. A., Mohammed, A. S., & Fakhrahmad, S. M. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112018 · Cited by 28
Sallam, M., Al-Mahzoum, K., Almutawaa, R. A., Alhashash, J. A., Dashti, R. A., AlSafy, D. R., Almutairi, R. A., & Barakat, M. M. (2024) · BMC Research Notes · doi:10.1186/s13104-024-06920-7 · Cited by 39
Disentangling patent quality: using a large language model for a systematic literature review
Schmitt, V. J. (2025) · Scientometrics · doi:10.1007/s11192-024-05206-w · Cited by 28
Kastrati, M., Imran, A. S., Hashmi, E., Kastrati, Z., Daudpota, S. M., & Biba, M. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110136 · Cited by 25
Aydemir, U. (2025) · JMIR Human Factors · doi:10.2196/71119 · Cited by 24
UniSA: Unified Generative Framework for Sentiment Analysis
Li, Z., Lin, T.-E., Wu, Y., Liu, M., Tang, F., Zhao, M., & Li, Y. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612336 · Cited by 32
Jung, H. S., Lee, H., Woo, Y. S., Baek, S. Y., & Kim, J. H. (2024) · PLoS ONE · doi:10.1371/journal.pone.0304680 · Cited by 21
GS-CBR-KBQA: Graph-structured case-based reasoning for knowledge base question answering
Li, J., Luo, X., & Lu, G. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125090 · Cited by 26
How can large language models assist with a FRAM analysis?
Sujan, M. A., Slater, D., & Crumpton, E. (2024) · Safety Science · doi:10.1016/j.ssci.2024.106695 · Cited by 22
How funny is ChatGPT? A comparison of human- and A.I.-produced jokes
Gorenz, D., & Schwarz, N. (2024) · PLoS ONE · doi:10.1371/journal.pone.0305364 · Cited by 26
Robust visual question answering via polarity enhancement and contrast
Peng, D., & Li, Z. (2024) · Neural Networks · doi:10.1016/j.neunet.2024.106560 · Cited by 20
LLM4Rerank: LLM-based Auto-Reranking Framework for Recommendations
Gao, J., Chen, B., Zhao, X., Liu, W., Li, X., Wang, Y., Wang, W., Guo, H., & Tang, R. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714922 · Cited by 87
Evaluating ChatGPT and Bard AI on Arabic Sentiment Analysis
Al-Thubaity, A. O., Alkhereyf, S., Murayshid, H. S., Alshalawi, N., Omirah, M., Alateeq, R., Almutairi, R. E., Alsuwailem, R. I., Alhassoun, M., & Alkhanen, I. (2023) · Arabic Natural Language Processing Conference (ArabicNLP) · doi:10.18653/v1/2023.arabicnlp-1.27 · Cited by 26
Jalali, N. A., & Hongsong, C. (2024) · Cluster Computing · doi:10.1007/s10586-024-04515-2 · Cited by 25
Hassani, H., & Silva, E. S. (2024) · Information · doi:10.3390/info15060291 · Cited by 23
Lu, R.-S., Lin, H.-C. K., Yang, Y.-C., & Chen, Y.-P. (2024) · Sustainability · doi:10.3390/su162411304 · Cited by 24
Crowd Sensing Intelligence for ITS: Participants, Methods, and Stages
Zhao, Y. B., Hu, C., Zhu, Z., Qiu, S., Chen, B., Jiao, P., & Wang, F. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3284046 · Cited by 19
Woźniak, S., & Kocoń, J. (2023) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw60847.2023.00108 · Cited by 23
Self-citationImproving Training Dataset Balance with ChatGPT Prompt Engineering
Kochanek, M., Cichecki, I., Kaszyca, O., Szydło, D., Madej, M., Jędrzejewski, D., Kazienko, P., & Kocoń, J. (2024) · Electronics · doi:10.3390/electronics13122255 · Cited by 20
Self-citationAI-Driven Financial Analysis: Exploring ChatGPT’s Capabilities and Challenges
Liu, L. X., Sun, Z., Xu, K., & Chen, C. (2024) · International Journal of Financial Studies · doi:10.3390/ijfs12030060 · Cited by 21
A survey on pragmatic processing techniques
Mao, R., Ge, M., Han, S., Wei, L., He, K., Zhu, L., & Cambria, E. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102712 · Cited by 24
Derner, E., Kučera, D., Oliver, N., & Zahálka, J. (2024) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2024.100088 · Cited by 25
SF-GPT: A training-free method to enhance capabilities for knowledge graph construction in LLMs
Sun, L., Zhang, P., Gao, F., An, Y., Li, Z., & Zhao, Y. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128726 · Cited by 21
Large Language Model Evaluation Criteria Framework in Healthcare: Fuzzy MCDM Approach
Alabool, H. M. (2025) · SN Computer Science · doi:10.1007/s42979-024-03533-6 · Cited by 22
Bi-DeepViT: Binarized Transformer for Efficient Sensor-Based Human Activity Recognition
Luo, F., Li, A., Khan, S., Wu, K., & Wang, L. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3526166 · Cited by 19
Zhong, Y., Lian, J., Huang, H., & Deng, H. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-13237-2 · Cited by 32
Gasaymeh, A.-M. M., Qbeita, A. A. A., Almohtadi, R., & Beirat, M. A. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1580310 · Cited by 32
Theory of Mind Abilities of Large Language Models in Human-Robot Interaction: An Illusion?
Verma, M., Bhambri, S., & Kambhampati, S. (2024) · IEEE/ACM International Conference on Human-Robot Interaction · doi:10.1145/3610978.3640767 · Cited by 51
Exploring Public Response to ChatGPT With Sentiment Analysis and Knowledge Mapping
Zhou, J., Liang, Z., Fang, Y., & Zhou, Z. (2024) · IEEE Access · doi:10.1109/access.2024.3386362 · Cited by 18
Choi, E. C., & Ferrara, E. (2024) · Companion Proceedings of the ACM Web Conference 2024 · doi:10.1145/3589335.3651910 · Cited by 18
Poretsky, E., Blake, V. C., Andorf, C. M., & Sen, T. Z. (2025) · Database · doi:10.1093/database/baaf011 · Cited by 15
Gong, Y., Xu, C., Luo, S., & Lin, J. (2025) · Scientific Reports · doi:10.1038/s41598-025-03298-9 · Cited by 32
Chun, J., & Elkins, K. (2023) · International Journal of Digital Humanities · doi:10.1007/s42803-023-00069-8 · Cited by 22
Saviano, M., Thomas, A., Prete, M. D., Verderese, D., & Sasso, P. (2025) · European Journal of Innovation Management · doi:10.1108/ejim-05-2024-0520 · Cited by 25
RWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.13048 · Cited by 17
Self-citationLaboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings
Wolfe, R., Slaughter, I., Han, B., Wen, B., Yang, Y., Rosenblatt, L., Herman, B., Brown, E., Qu, Z., Weber, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658966 · Cited by 24
Zhao, Y., Tang, S., Zhang, H., & Lyu, L. (2025) · Journal of Retailing and Consumer Services · doi:10.1016/j.jretconser.2025.104400 · Cited by 28
Alruqi, T. N., & Alzahrani, S. M. (2023) · AI · doi:10.3390/ai4030035 · Cited by 32
Awal, M. R., & Asaduzzaman (2024) · Higher Education Skills and Work-based Learning · doi:10.1108/heswbl-09-2023-0249 · Cited by 19
Data science through natural language with ChatGPT’s Code Interpreter
Ahn, S. (2024) · Translational and Clinical Pharmacology · doi:10.12793/tcp.2024.32.e8 · Cited by 16
Usage of Large Language Model for Code Generation Tasks: A Review
Stefano Bistarelli, B., Fiore, M., Mercanti, I., & Mongiello, M. (2025) · SN Computer Science · doi:10.1007/s42979-025-04241-5 · Cited by 29
Suryanto, T. L. M., Wibawa, A. P., Hariyono, H., & Nafalski, A. (2023) · International Journal of Robotics and Control Systems · doi:10.31763/ijrcs.v3i4.1195 · Cited by 29
Gams, M., & Kramar, S. (2024) · Journal of Computer and Communications · doi:10.4236/jcc.2024.123014 · Cited by 15
Evolving Generative AI: Entangling the Accountability Relationship
Elliott, M. T. J., Deepak, P., & MacCárthaigh, M. (2024) · Digital Government Research and Practice · doi:10.1145/3664823 · Cited by 17
The Evolution of Learning: Assessing the Transformative Impact of Generative AI on Higher Education
Krause, S., Panchal, B. H., & Ubhe, N. (2024) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-97-9255-9_24 · Cited by 35
LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs
Wu, T., Zhu, H., Albayrak, M., Axon, A., Bertsch, A., Deng, W., Ding, Z., Guo, B., Gururaja, S., Kuo, T.-S., et al. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3706690 · Cited by 59
A Data-Driven Approach for the Identification of Features for Automated Feedback on Academic Essays
Abbas, M., van Rosmalen, P., & Kalz, M. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3320877 · Cited by 16
Fang, F., Tian, Y., Dai, D., & Xing, S. (2024) · Remote Sensing · doi:10.3390/rs16061013 · Cited by 14
Macanovic, A., & Przepiorka, W. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02381-9 · Cited by 20
Faraz, A., Ahsan, F., Mounsef, J., Karamitsos, I., & Kanavos, A. (2024) · Information · doi:10.3390/info15040233 · Cited by 20
Haase, J., Kremser, W., Leopold, H., Mendling, J., Onnasch, L., Technische Universität Berlin, G., & Plattfaut, R. (2024) · Communications of the Association for Information Systems · doi:10.17705/1cais.05421 · Cited by 15
Doctoral Programme in Education Science Universitas Pendidikan Ganesha, S., & Turmuzi, M. (2024) · International Journal of Information and Education Technology · doi:10.18178/ijiet.2024.14.5.2097 · Cited by 18
ChatGPT, Help! I Am in Financial Trouble
Schlosky, M. T. T., Karadas, S., & Raskie, S. (2024) · Journal of risk and financial management · doi:10.3390/jrfm17060241 · Cited by 14
StuGPTViz: A Visual Analytics Approach to Understand Student-ChatGPT Interactions
Chen, Z., Wang, J., Xia, M., Shigyo, K., Liu, D., Zhang, R., & Qu, H. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456363 · Cited by 37
Yang, Y., Li, J., Guo, J., Pang, P. C., Wang, Y., Yang, X., & Im, S. (2025) · IEEE Access · doi:10.1109/access.2025.3549733 · Cited by 16
Exploration of applications with ChatGPT for green supply chain management
Zhou, L., Shi, X., Wang, Z., Ma, C., & Gao, L. (2025) · Annals of Operations Research · doi:10.1007/s10479-025-06713-6 · Cited by 16
A Transformer-Based Educational Virtual Assistant Using Diacriticized Latin Script
Lâm, K. N., Nguy, L. H., Le, V. L., & Kalita, J. K. (2023) · IEEE Access · doi:10.1109/access.2023.3307635 · Cited by 15
The Influence of ChatGPT on Student Learning and Academic Performance
Ayman, S. E., El-Seoud, S. A., Nagaty, K., & Karam, O. H. (2023) · International Conference on Computer and Applications (ICCA) · doi:10.1109/icca59364.2023.10401713 · Cited by 23
Rane, N. L. (2024) · International Journal of Artificial Intelligence and Machine Learning · doi:10.51483/ijaiml.4.1.2024.22-47 · Cited by 68
Wang, Z., Wang, Z., & Deng, R. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2331855 · Cited by 13
Exploring the capabilities of ChatGPT in women’s health: obstetrics and gynaecology
Bachmann, M., Duta, I., Mazey, E., Cooke, W. R., Vatish, M., & Jones, G. D. (2024) · npj Women s Health · doi:10.1038/s44294-024-00028-w · Cited by 14
Use and impact of ChatGPT on undergraduate engineering students: A case from the Philippines
Giray, L., & Aquino, R. (2024) · Internet Reference Services Quarterly · doi:10.1080/10875301.2024.2384028 · Cited by 18
Metadata
- Authors
- Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., Kocoń, A., Koptyra, B., Mieleszczenko-Kowszewicz, W., Miłkowski, P., Oleksy, M., Piasecki, M., Radliński, Ł., Wojtasik, K., Woźniak, S., & Kazienko, P.
- Affiliations
- Wrocław University of Science and Technology (PL)
- Corresponding author
- Kocoń, Jan
- ORCID
- Venue
- Information Fusion
- Publisher
- Elsevier BV
- Published
- 2023-11
- Volume
- 99
- Pages
- 101861
- ISSN
- 1566-2535 (ISSN-L)
- OpenAlex
- W4379259169
- Semantic Scholar
- 5848737f78397f72ceae2ba6f3419a6a8502b8ba
- Cited by
- 758
- References
- 105
- Citations by year
- 2026: 95 · 2025: 234 · 2024: 313 · 2023: 115
- FWCI
- 71.55
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Information Fusion (publishedVersion, cc-by)
- SSRN Electronic Journal (acceptedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Ministerstwo Edukacji i Nauki
- Politechnika Wrocławska
- HORIZON EUROPE Framework Programme
- European Regional Development Fund (01-00-0288/22, POIR.01.01, POIR.04.02.00-00C002/19)
- European Commission Marie Sklodowska-Curie Actions (101086321)
- Department of Artificial Intelligence, Korea University
- Narodowe Centrum Nauki (2021/41/B/ST6/04471)
- European Research Executive Agency
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- ChatGPT, GPT-4, large language models, automated assessment, sentiment analysis, emotion recognition, stance detection, word sense disambiguation, linguistic acceptability, question answering, few-shot prompting, algorithmic bias, human-machine interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
