The DOI matches its Crossref record; 80 of the 105 references it cites were found in a registry or an index; none of the 15 sources its claims cite is checked for support yet.
ChatGPT: Jack of all trades, master of none
Open accessAbstract
OpenAI has released the Chat Generative Pre-trained Transformer (ChatGPT) and revolutionized the approach in artificial intelligence to human-model interaction. The first contact with the chatbot reveals its ability to provide detailed and precise answers in various areas. Several publications on ChatGPT evaluation test its effectiveness on well-known natural language processing (NLP) tasks. However, the existing studies are mostly non-automated and tested on a very limited scale. In this work, we examined ChatGPT’s capabilities on 25 diverse analytical NLP tasks, most of them subjective even to humans, such as sentiment analysis, emotion recognition, offensiveness, and stance detection. In contrast, the other tasks require more objective reasoning like word sense disambiguation, linguistic acceptability, and question answering. We also evaluated GPT-4 model on five selected subsets of NLP tasks. We automated ChatGPT and GPT-4 prompting process and analyzed more than 49k responses. Our comparison of its results with available State-of-the-Art (SOTA) solutions showed that the average loss in quality of the ChatGPT model was about 25% for zero-shot and few-shot evaluation. For GPT-4 model, a loss for semantic tasks is significantly lower than for ChatGPT. We showed that the more difficult the task (lower SOTA performance), the higher the ChatGPT loss. It especially refers to pragmatic NLP problems like emotion recognition. We also tested the ability to personalize ChatGPT responses for selected subjective tasks via Random Contextual Few-Shot Personalization, and we obtained significantly better user-based predictions. Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI. Our results provide the basis for a fundamental discussion of whether the high quality of recent predictive NLP models can indicate a tool’s usefulness to society and how the learning and validation procedures for such systems should be established.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper evaluates ChatGPT on 25 analytical NLP tasks and GPT-4 on five subsets, automating prompts and analyzing more than 49k responses. It finds ChatGPT never reaches SOTA performance, with an average loss of about 25%, and that losses are larger for harder, pragmatic, and especially emotion-related tasks.
Key findings
- ChatGPT underperformed SOTA on every task: average SOTA quality was 73.7% versus ChatGPT’s 56.5%, with an average loss of 25.5%; ChatGPT was also less stable (std. dev. 23.3% vs. 16.7%). Excluding eight emotion-related tasks, average loss fell to 12.8% (ChatGPT 69.7% vs. SOTA 80.0%).
- Loss correlated with task difficulty (Pearson 0.46); pragmatic tasks were harder on average (32.3% difficulty) than semantic tasks (15.6%), and emotion tasks had an average loss of 52.59% (ChatGPT 28.44% vs. SOTA 60.28%).
- Random Contextual Few-Shot Personalization improved user-based predictions: GoEmoPer3 was 4.08 percentage points better than GoEmoPer0, while AggressionPer and UnhealthyPer showed higher ChatGPT loss than their non-personalized counterparts by 3.25 and 12.55 percentage points, respectively.
- GPT-4 had small loss on semantic tasks (0.9%–6.4% for TextEntail and WNLI) but still high loss on emotional tasks (46.3%–56.3%); ChatGPT still outperformed GPT-4 on GoEmo and PolEmo. Qualitative analysis found bias, moralizing refusals, non-determinism, and cases where experts accepted ChatGPT answers over human labels (e.g., Unhealthy: 81% ChatGPT vs. 43% human).
Methods
- Tested 25 NLP datasets/tasks (23 English, 2 Polish), split into semantic and pragmatic categories, including sentiment, emotion, offensiveness, stance, WSD, NLI, and QA; some tasks used personalized user context.
- Automated prompting via an unofficial PyGPT API using up to 20 accounts, with manual post-processing; the paper reports over 38,000 prompts in the setup and more than 49k responses in the abstract.
- Compared ChatGPT to SOTA models by rerunning available source code or using reported metrics, using F1 Macro or accuracy and computing Loss = 100% × (SOTA − ChatGPT) / SOTA; GPT-4 was tested on five selected subsets.
Limitations
- Stated: prompts may not be strict or precise enough; post-processing was sometimes required; ChatGPT produced unexpected out-of-range responses; internal policy and biases affected answers; context was generally respected but offered limited control; efficiency and costs were constrained by lack of an official API, with about 2–3k prompts per day achieved.
- Stated: the difficulty analysis is limited because only 25 tasks were considered; GPT-4 testing was restricted by access limits (25 prompts per 3 hours).
- Evident from the text: dataset availability for ChatGPT training was assessed manually and is uncertain; qualitative expert evaluation covered 100 randomly selected cases per dataset and involved subjective judgments; ChatGPT showed non-determinism, with results varying across launches.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
98 Own result · 37 Own interpretation · 34 No basis given · 11 Own method · 10 Not checked yet
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For GPT-4 model, a loss for semantic tasks is significantly lower than for ChatGPT.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We showed that the more difficult the task (lower SOTA performance), the higher the ChatGPT loss.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Simultaneously, ChatGPT was less stable: the standard deviation of its performance was 23.3% compared to only 16.7% for the SOTA solutions.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“because for all of these datasets, we observe a significant drop in quality relative to state-of-the-art models.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It especially refers to pragmatic NLP problems like emotion recognition.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also tested the ability to personalize ChatGPT responses for selected subjective tasks via Random Contextual Few-Shot Personalization, and we obtained significantly better user-based predictions.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It is usually greater for semantic tasks rather than for pragmatic ones, which is related to the task difficulty, see Sec. 6.2.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our comparison of its results with available State-of-the-Art (SOTA) solutions showed that the average loss in quality of the ChatGPT model was about 25% for zero-shot and few-shot evaluation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“yet it has them in topics regarding ideological views.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 46
- Probably not a substantial claim. 70
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The crucial finding from our studies is that the ChatGPT performance is always lower than the SOTAmethods (loss>0) in all the tasks considered.”
- Also stated in: § 6.1. Jack of all trades, master of none · p. 10 · plain
“It means that ChatGPT never reached the level of the best existing models.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 6.1. Jack of all trades, master of none · p. 10 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The loss for most tasks did not exceed 25%. It was greater only for three problems: GoEmotions, PolEmo, and TweetEmoji.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The average quality of SOTA methods was at 73.7%, whereas ChatGPT was at only 56.5%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while GPT-4 achieved significantly higher scores in the remaining three semantic tasks with even a 9.9 p.p. increase for the WNLI task.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While solving mathematical calculations (SQuAD), the highest loss was among semantic tasks: 23.7%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, pragmatic tasks are more difficult (average difficulty = 32.3%), while the average difficulty for semantic tasks is only 15.6%.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also noticed that when it is unable to perform a task on the provided example, it will refuse to do so and provide an explanation why”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Those situations are usually solved by supplementing the model with information.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It comes especially from the emotional tasks, which are pragmatic and very difficult (average 39.7%).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the case of mathematical reasoning, it can provide a whole explanation of how it reached a certain outcome and provide only the answer without explanation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When exploiting the possibilities of ChatGPT, we can see that it can perform various tasks, including recognizing generalized and personalized dimensions of Natural Language Processing, answering questions where a generous amount of domain knowledge is required, or even writing lines of code in the programming language of choice.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can also observe that the loss is correlated with the task difficulty; see Fig. 7. The Pearson correlation coefficient between difficulty and loss is equal to 0.46.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When conducting experiments regarding tasks such as humor recognition or offensiveness detection, we have stumbled upon output that not only refuses to answer whether something is or is not funny but also sends a moralizing message with an irritated tone.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ChatGPT can also return a number as a category indicator or whole output in the JSON format.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In such a case, the average loss is reduced by as much as half, from 25.5% to 12.8%; the difference in performance drops from 17.2pp to 10.3pp.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If we discard all eight emotion-related tasks (ids: 12-17, 20-21), the average SOTA performance reaches 80%(increase by 6.3pp), but ChatGPT improves much more: by 13.2pp, up to 69.7%.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ChatGPT will only generate the content if the user adequately motivates it with the scientific goals.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, in the case of the ChatGPT model, prompt tuning is technically unavailable”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“as it has happened in the case of ClarinEmo B.12, where the model stated that all provided texts are legal and financial statements.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Anyway, the differences in performance are not significant.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Onthe other hand, ChatGPT demonstrated its instability and tendency towards non-determinism.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It also refers to the necessary manual correction of typos surprisingly provided by ChatGPT”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, prompts themselves and the sequence of prompts provide limited control over them.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“So it evaluates the main content but ignores the broader context (see Chat 64).”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Therefore, some responses are locked due to the OpenAI policy.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“e.g., the unlisted emotion satisfaction has been delivered:”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In this example, the sentence was interpreted as not aligned with society's standards.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“## Who is right? (expert rate)”
- Also stated in: § Expected · p. 41 · plain
“Evaluation result human - not correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Also stated in: § Expected · p. 41 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“## Who is right? (expert rate)”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluation result human - not correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“## Who is right? (expert rate)”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“human - correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluation result human - not correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The annotator has evaluated the below message as expressing gratitude, whereas ChatGPT regards it as neutral (Chat 58)”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“3. ChatGPT flattens the message, partially ignoring the metatext.”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, ChatGPT explicitly distances itself from judging people.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ChatGPT labeled as aggressive only 11 texts from the WikiDetox Aggression dataset labeled by people as non-aggressive, while the opposite decision was taken 207 times.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluation result human - not correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Restrictive or wrongly implemented policy will influence the user experience of the ChatGPT by not answering all user's prompts exaggeratedly judging many questions not along with OpenAI rules:”
hedgeddirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“human - correct, ChatGP - not correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“when asked which language phenomena demonstrate the enthusiasm or positive sentiment in the text, it gives a list containing such phenomena as the use of superlatives or lack of negative language”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Context is generally respected by ChatGPT.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“human - correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The problem of controversial and ambiguous utterances is solved by ChatGPT's demonstration of a lower level of confidence”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“## Who is right? (expert rate)”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“human - not correct , ChatGPT - not correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“## Who is right? (expert rate)”
plainSingle case- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As mentioned in the previous section, ChatGPT tends to find negative connotations in the given text.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63:”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Similarly, ChatGPT justifies the negative sentiment of the news about the ban on naming streets after Fidel Castro, Chat 66:”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Internal policy can strongly influence the answers received.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All these tasks are related to a very subjective problem of emotional perception and individual interpretation of the content.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“In cases where the performance of ChatGPT was compared to other solutions (e.g. [26, 30, 41]), it worked on a comparable level to the competitor but not outperforming any major SOTA solutions.”
What the sources say
- Not checked yetcites [26] Kung, 2022
- Not checked yetcites [30] Goyal, 2022
- Not checked yetcites [41] Jiao, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“There were a couple of large-scale evaluations published recently [46, 47] and both of them indicate that while the performance of ChatGPT is significant it doesn't outperform the SOTA solutions with the exception of sentiment analysis task in case of [46] which suggests it is a significant area for further research.”
What the sources say
- Not checked yetcites [46] Amin, 2023
- Not checked yetcites [47] Bang, 2023
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
In the paper
“ChatGPT's successor, the GPT-4 [19], is most likely an even larger model that can additionally receive not only text but also images as input.”
What the sources say
- Not checked yetcites [19] OpenAI, 2023
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Language models are prone to generating responses containing human-like biases as well as presenting moral and ethical stances [23].”
What the sources say
- Not checked yetcites [23] Schramowski, 2022, Large pre-trained language models contain human-like biases of what is right and wrong to do, Nat. Mach. Intell. source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even though the creators of ChatGPT secure the model against answers that are offensive, there are still multiple methods that may reveal its hidden biases.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nevertheless, we can state that ChatGPT performs pretty well on all tasks except emotional ones.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, the number of cases that required post-processing was relatively small (column #Post-processed in Tab. 1). For most tasks (16), the contribution of such texts was less than 5%. Only for Aggression, SQuAD, Unhealthy, and ReAding, it exceeded 15%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Manual post-processing correctly extracted and mapped ChatGPT's raw outputs to the intended labels.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Simultaneously, its achievements are always below SOTA but usually not so much.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such results prove that ChatGPT is Jack of all trades, master of none .”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, using the prompts directly as humans designed them implicitly allows us to evaluate models' language comprehension abilities.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We emphasize that the results for GPT-4, as for ChatGPT, could significantly differ, if distinct prompt schemes are compared, see Sec. 6.6.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“If possible, we launched our models equivalent to SOTA solutions since the setup (especially data split) was often different than in the original paper. For that purpose, we usually utilized source codes published by the authors. Unfortunately, it was impossible for some tasks, so we exploited the performance results provided in the original paper.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: The SOTA results taken from original papers are comparable to the paper's own setup despite differences in data split and evaluation.
In the paper
“Borji [105] conducted a systematic review of the typical categories of ChatGPT failures.”
What the sources say
- Not checked yetcites [105] Korczyński, 2022, Compression methods for transformers in multidomain sentiment analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The above errors are derived from both incorrect reasoning in terms of psychological reasoning and bias and discrimination.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This analysis, however, requires further investigations since the number of the tasks considered (25) still remains relatively small.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is observable that semantic tasks (blue crosses) are rather easy; hence, their ChatGPT loss is relatively small.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The annotations of six independent annotators were aggregated to label each sentence with all potential options, using the label when at least two annotators agreed on it.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using the glosses to explain senses to a language model implicitly tests its language comprehension abilities.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“It is our new dataset that has not yet been published. We exploited this dataset to ensure that ChatGPT was not trained on it.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: ChatGPT was not trained on the ClarinEmo dataset.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and the only way to verify prompt relevance is to evaluate its performance directly in the downstream task.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“If available, we tried to validate ChatGPT using one measure F1 Macro, which is commonly acceptable for imbalanced data, Tab. 2.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In the case of CoLa, WNLI, WordContext, and MathQA, we had to rely on the accuracy, as it was the only one presented in the reference paper; we could not replicate their studies and calculate our measures. WNLI and WordContext have their two classes balanced, so it is not an issue.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“which further implies possible consequences regarding clashes with fake news.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It may suggest that providing some answerable questions helps it detect unanswerable ones with the same context.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Demonstration-based personalization included in our prompts can be treated as similar to few-shot learning, even an extended context to a given prompt.”
plaincorrelationaln = 300- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Hagendorff [104] drew attention to the fact that chatbot ethics can be a subject of debate in fairness, nondiscrimination, and justice.”
What the sources say
- Not checked yetcites [104] Borji, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, the raw text provided by ChatGPT may require extraction of crucial information, e.g., final labels.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Information about the distinction for a particular footballer is neutral.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“1. Prompts may not be strict and precise enough.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“2. ChatGPT seems to have no regard for individuals, instead judging situations.”
hedgedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, this often leads to mistakes”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“6. ChatGPT presents the sense of common human morality.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ChatGPTrequires prompts in natural language, which is - by nature - not structured and can be different for different users and tasks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“A small difference in these two values proves a similar distribution of classes in both sets, thus, a good stratification of sampling.”
- Also stated in: § Table 8 · p. 28 · boosted
“A marginal difference in entropy of output real values for test or dev set (column #Test in Tab. 1) and within the set used by us (column #Used in Tab. 1) demonstrates a good stratification of the used set selection.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Table 8 · p. 28 · boosted
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It also depends on the prompting quality and correctness.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its sentiment, however, can be both positive and negative. It depends on the sympathies of the recipient sympathies regarding specific footballers.”
plainconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT evaluates rather situations than participants, but words refer to people, which lead to a specific, predetermined assessment”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Analysis was conducted by experts who are specialists trained in the recognition of emotions in the text. One of them is a psychologist and another is a linguist, both are experienced annotators.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It also provides a unique selfexplanation capability that facilitates human understanding and adaptation to the expected outcome.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, we are dealing with self-explanatory artificial intelligence, which is a part of eXplainable Artificial Intelligence (XAI)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It includes various biases and political correctness incorporated into the general ChatGPT model.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper relies on this without support.
The 25 selected NLP tasks are representative of typical NLP analytical tasks.
“We tested ChatGPT on 25 tasks focusing on solving common NLP problems and requiring analytical reasoning, Tab. 1.”
The paper relies on this without support.
ChatGPT's self-reports about whether it used a dataset for training are reliable.
“Additionally, we asked ChatGPT whether or not the dataset was used for training.”
The paper relies on this without support.
ChatGPT was not trained on the ClarinEmo dataset.
“It is our new dataset that has not yet been published. We exploited this dataset to ensure that ChatGPT was not trained on it.”
The paper relies on this without support.
The unofficial PyGPT API and the use of up to 20 accounts produced outputs equivalent to the official ChatGPT service.
“Without an official API, we modified and used an unofficial API called PyGPT 5 , written in Python. During the research, we exploited up to 20 accounts to gather data regarding 25 datasets.”
The paper relies on this without support.
The SOTA results taken from original papers are comparable to the paper's own setup despite differences in data split and evaluation.
“If possible, we launched our models equivalent to SOTA solutions since the setup (especially data split) was often different than in the original paper. For that purpose, we usually utilized source codes published by the authors. Unfortunately, it was impossible for some tasks, so we exploited the performance results provided in the original paper.”
The paper relies on this without support.
Manual post-processing correctly extracted and mapped ChatGPT's raw outputs to the intended labels.
“This imposes a necessity to check what happened and extract answers from ChatGPT output manually. The next step is to cast the resulting outputs to the correct labels in the dataset.”
The paper relies on this without support.
The performance of modern language models such as T5, GPT-3, and ChatGPT heavily relies on the quality of task-specific prompts.
“The performance of modern language models, such as T5, GPT-3, and ChatGPT, heavily relies on the quality of task-specific prompts.”
The paper relies on this without support.
The prompt-based learning paradigm requires careful prompt engineering and prompt tuning.
“The prompt-based learning paradigm requires careful prompt engineering and prompt tuning.”
The paper relies on this without support.
Most tasks require a prompt that enables the model to choose a certain value from the provided options.
“Most tasks require a prompt that enables the model to choose a certain value from the provided options.”
The paper relies on this without support.
The annotations in the selected 4 datasets were of a more or less subjective nature, and detailed guidelines were not necessary to achieve high interannotator agreement.
“The annotations in the selected 4 datasets were of a more or less subjective nature, and for this reason, it was not necessary to create detailed guidelines in order to achieve high interannotator agreement”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In a large study of ChatGPT that included 25 tasks and over 48,000 prompts, the tool was found to perform reasonably well when conducting quantitative analysis, with a somewhat poorer performance with tasks that require more reasoning ( Kocoń et al., 2023).”
From Generative AI for corpus approaches to discourse studies: A critical evaluation of ChatGPT (2024) · cites this paper as Kocoń et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This underperformance by the chat model is consistent with the findings by Kocon et al. [39].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [39]
In the citing paper
“Similarly, LLMs’ performance in logic and deductive tasks is often poor (64, 65), so using them for analysis may lead to false conclusions.”
From How should the advancement of large language models affect the practice of science? (2025) · cites this paper as [64]
In the citing paper
“Kocoń et al. (76) found that LLMs may struggle with complex, subjective tasks such as emotion recognition, and Huang et al. (18) found that LLMs can reinforce lay perspectives when detecting implicit hate speech and may thus mislead nonexperts with their explanations in cases where they make incorrect decisions.”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [76]
In the citing paper
“However, this perspective of LLMs, as a convenient and comprehensive tool in psychological text analysis, is challenged by recent critiques emphasizing their limitations, such as inconsistencies in text annotations, difficulties in explaining complex constructs such as implicit hate speech, and a potential lack of depth in specialized or sensitive areas (18, 76, 77).”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [76]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| 2 | AggressionPer | Pragmatic | F1 Macro | Our | 72.57 | 81.03 | 8.46 | 19.97 | 10.44 |”
Reported 19.97 · recomputed 18.97
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 6.3. Random Contextual Few-Shot Personalization · p. 12
“| 24 | ReAding | Semantic | F1 Macro | Our | 76.36 | 84.71 | 8.35 | 15.29 | 9.86 |”
§ Prompt · p. 29
“| 24 | ReAding | 76.36 | 76.34 | - | - | 84.71 | - |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 6.3. Random Contextual Few-Shot Personalization · p. 12
“| 8 | TextEntail | Semantic | F1 Macro | Paper | 88.09 | 92.10 | 4.01 | 7.90 | 4.35 |”
§ Prompt · p. 29
“| 8 | TextEntail | 88.09 | 87.88 | - | - | 92.1 | - |”
Citations
Cites
1 retracted
RETRACTED: WELMSD – word embedding and language model based sarcasm detection
Kumar, P., & Sarin, G. (2022) · Online Information Review · doi:10.1108/oir-03-2021-0184 · Cited by 17 · cited 1×
“3%) 995 2 20137 /19643 5 Sarcasm P EN Humor recognition No Binary classification Sarcasmania [67] / [68] 3 Yes 5967 1000 10 61 (6.”
BackgroundRetractedBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kilgarriff, A. (2000) · Language · doi:10.2307/417141 · Cited by 11,529
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
A survey of deep neural network architectures and their applications
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.038 · Cited by 3,239
Research works printed but not linked: 6
- Johnson, 2016, Supervised and semi-supervised text categorization using LSTM for region embeddings, vol. 48, 526
- Gillioz, 2020, Overview of the transformer-based models for NLP tasks, vol. 21, 179
- Radford, 2018, Improving language understanding by generative pre-training, OpenAI
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI
- Ngo, 2022, StudEmo: A non-aggregated review dataset for personalized emotion recognition, 46
- Fellbaum, 1998, Wordnet: An electronic lexical database, Comput. Linguist., 292
Cited by
Sorted by citations
Items 501–600 of 724
Szczęsny, A., Markiewicz, M., Radliński, Ł., & Kazienko, P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-97635-3_7 · Cited by 2
Self-citationOptimized dimension-reduction algorithm on input layer for diffractive neural networks
Li, R., Ma, Q., Gu, Z., Ning, Y. M., Gao, X., & Cui, T. J. (2025) · APL Photonics · doi:10.1063/5.0279686 · Cited by 1
Dong, W., Wang, W., Han, X., Huang, J., Li, L., & Huang, Y. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110789 · Cited by 2
Wang, W., Li, Y., & Wu, Y. J. (2025) · Springer proceedings in complexity · doi:10.1007/978-3-031-78623-5_12 · Cited by 1
Comparison of Generative Artificial Intelligence and Student-Generated Veterinary Handouts
Gibson, R., Tomberlin, S. S., Mackay, L., & Foradori, C. D. (2025) · Journal of Veterinary Medical Education · doi:10.3138/jvme-2025-0003 · Cited by 2
Student chatbot use and perceptions in a course assignment: Comparing two geomatics courses
Hochmair, H. H. (2025) · Geomatica · doi:10.1016/j.geomat.2025.100079 · Cited by 1
Nonlinear transformation of probabilities by large language models
Hintze, A., Bisht, C., Schossau, J., & Hertwig, R. (2025) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2025.100227 · Cited by 1
Classifying illicit dark web content through zero-shot prompting: An empirical study with GPT models
Domínguez‐Díaz, A., de‐Marcos, L., Prado-Sánchez, V.-P., et al. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104476 · Cited by 1
Rahman, M. I., Akter, M. S., & Sayem, S. M. (2025) · Discover Public Health · doi:10.1186/s12982-025-01233-x · Cited by 2
Dake, D. K., & Gbagbo, F. Y. (2025) · Discover Education · doi:10.1007/s44217-025-01074-5 · Cited by 3
Bridging AI and Human Knowledge: Towards a Deeper Understanding of Stack Overflow and ChatGPT
Swaraj, A., & Kumar, S. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3757002 · Cited by 1
One-Sided Parasociality with AI
Degen, J. L. (2025) · Palgrave Studies in Cyberpsychology. · doi:10.1007/978-3-032-07438-6_5 · Cited by 2
Zhong, L., & Ma, X. (2026) · British Educational Research Journal · doi:10.1002/berj.70098 · Cited by 2
Building an emotion lexicon for Serbian using curated language resources
Šošić, M., Graovac, J., & Stanković, R. M. (2026) · Language Resources and Evaluation · doi:10.1007/s10579-025-09894-5 · Cited by 3
Nuortimo, K., Härkönen, J., & Breznik, K. (2026) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2026.124553 · Cited by 3
The pragmatic competence of ChatGPT. An AI-assisted research project
Vasilescu, A. (2025) · Revue roumaine de linguistique · doi:10.59277/rrl.2025.3-4.15 · Cited by 1
Altameemi, Y. M. (2026) · Cogent Arts and Humanities · doi:10.1080/23311983.2025.2592697 · Cited by 1
Aiyappa, R., Senthilmani, S., An, J., Kwak, H., & Ahn, Y. (2026) · PeerJ Computer Science · doi:10.7717/peerj-cs.3540 · Cited by 14
Li, Y., Xia, Y., Shi, H., He, L., & Li, Y. (2026) · The North American Journal of Economics and Finance · doi:10.1016/j.najef.2026.102604 · Cited by 2
Zhou, W., He, D., Jin, Z., Wei, Z., Sun, H., Wu, J., Zhuang, Y., Qin, B., & Wu, Q. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.114904 · Cited by 2
Wei, Y., Li, X., Huang, Z., Lin, Z., & Zhang, N. (2026) · Computers in Industry · doi:10.1016/j.compind.2026.104462 · Cited by 1
Humanizing Online Learning: The Role of AI+XR-Based Virtual Tutors in Adaptive Education
Cardoso, A., de Siqueira, A. G., Martins, V. F., Neto, J. S. D. O., Eliseo, M. A., & Silveira, I. F. (2025) · Congresso Latino-Americano de Software Livre e Tecnologias Abertas (Latinoware) · doi:10.5753/latinoware.2025.16483 · Cited by 1
Pihulski, D., & Kocoń, J. (2025) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw69685.2025.00261 · Cited by 1
Self-citationSantos, J. M. R. C. A., & Fernandes, G. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.03.136 · Cited by 1
ChatGPT estimates of the quality of published conference papers from their titles and abstracts
Thelwall, M. A. (2026) · Data Technologies and Applications · doi:10.1108/dta-03-2025-0205 · Cited by 4
Can ChatGPT reciprocate impoliteness? The AI moral dilemma
Tantucci, V., & Culpeper, J. (2026) · Journal of Pragmatics · doi:10.1016/j.pragma.2026.03.008 · Cited by 4
Yao, Q., Ban, R., Xiang, L., Wang, X., Hu, A., & Bing, H. (2026) · Applied Energy · doi:10.1016/j.apenergy.2026.128058 · Cited by 3
Wang, F., Zhou, J., & Zheng, X. (2026) · Journal of Construction Engineering and Management · doi:10.1061/jcemd4.coeng-18215 · Cited by 2
Automated Coding of Communication Data Using LLM: Consistency across Subgroups
Hao, J., Cui, W., Kyllonen, P. C., & Kerzabi, E. (2026) · Journal of Educational Measurement · doi:10.1111/jedm.70049 · Cited by 2
Fan, J., Chen, D., Liu, Y., et al. (2026) · International Journal of Remote Sensing · doi:10.1080/01431161.2026.2691981 · Cited by 1
In Which Fields Do ChatGPT Scores Align More Closely with Research Quality than Do Citation Rates?
Thelwall, M. A. (2026) · Journal of Data and Information Science · doi:10.1515/jdis-2026-0058 · Cited by 17
Setiawan, R. N., Hidayatulloh, N. W., Doko, V. E. O. E., et al. (2026) · F1000Research · doi:10.12688/f1000research.190739.1 · Cited by 1
Bielaniewicz, J., & Kazienko, P. (2023) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw60847.2023.00098 · Cited by 1
Self-citationApplications Areas, Possibilities, and Limitations of ChatGPT
Qamar, R., & Zardari, B. A. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-6824-4.ch004 · Cited by 1
EmoBlend Fusion: Leveraging Handcrafted and Deep Features for Emotion Detection
Anzum, F., & Gavrilova, M. L. (2024) · IEEE International Conference on Human-Machine Systems (ICHMS) · doi:10.1109/ichms59971.2024.10555818 · Cited by 1
Faceless Adversary, Feckless Colleague: The Many Sides of ChatGPT
Bird, W. H. (2024) · Western Canadian Conference on Computing Education · doi:10.1145/3660650.3660656 · Cited by 2
A Multi-Task Learning Framework for Reading Comprehension of Scientific Tabular Data
Xu, Y., Zhang, M., Fan, J., Luo, Z., & Yang, Y. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00285 · Cited by 1
Butt, A. A., babar, E. T. R., Menekşe, M., & Alhaddad, A. (2024) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--47060 · Cited by 1
Machine Learning Method for Detecting Propaganda in Twitter Texts
Mynzar, B., Stetsenko, I. V., Gordienko, Y. G., et al. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-67348-1_15 · Cited by 1
Social media versus surveys: A new scalable approach to understanding legislators' discourse
González‐Rostani, V., Incio, J., & Lezama, G. (2024) · Legislative Studies Quarterly · doi:10.1111/lsq.12481 · Cited by 1
Kos, M. O., & Grošelj, D. (2024) · doi:10.18690/um.pf.5.2024.6 · Cited by 1
GenAI: Unlocking Sustainability Insights and Driving Change in Fintech
Gopal, S., & Pitts, J. (2024) · Sustainable finance · doi:10.1007/978-3-031-74418-1_9 · Cited by 2
Evaluating the Acceptance of Enhanced Generative AI Services
Ahmed, W., Sentosa, I., Hizam, S. M., et al. (2023) · doi:10.1109/cdics61497.2023.00022 · Cited by 1
Inventory Management in Online Store—Strategy Overview
Chodak, G. (2024) · Springer eBooks · doi:10.1007/978-3-031-78716-4_1 · Cited by 1
Relevant emotion ranking with topic-enhanced emotion transition
Zhang, L., Zhang, X., & Zhou, D. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113015 · Cited by 1
Testing the limits of large language models in debating humans
Flamino, J., Modi, M. S., Szymanski, B. K., Cross, B., & Mikolajczyk, C. (2025) · Scientific Reports · doi:10.1038/s41598-025-98378-1 · Cited by 1
de Paredes, P. G., Ramadier, T., Baron, M. P., LeBlanc, A., & Després, C. (2025) · Cities & Health · doi:10.1080/23748834.2025.2496557 · Cited by 1
Evolving Deep Architectures: A New Blend of CNNs and Transformers Without Pre-training Dependencies
Kiiskilä, M., & Kiiskilä, P. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-66694-0_10 · Cited by 1
Is ChatGPT capable of solving classical Communications and Networking problems?Preprint
Hernández, J. A., Conde, J., Reviriego, P., & de Arcaute, G. M. R. (2023) · doi:10.36227/techrxiv.23727174 · Cited by 1
Is ChatGPT capable of solving classical Communications and Networking problems?Preprint
Hernández, J. A., Conde, J., Reviriego, P., & de Arcaute, G. M. R. (2023) · doi:10.36227/techrxiv.23727174.v1 · Cited by 1
Is it possible for ChatGPT to mimic human annotator?Preprint
Kaszyca, O., Kazienko, P., Kocoń, J., Cichecki, I., Kochanek, M., & Szydło, D. (2023) · doi:10.36227/techrxiv.170259175.55207879/v1 · Cited by 1
Self-citationCan innovative prompt engineering with ChatGPT address imbalances in machine learning datasets?Preprint
Kochanek, M., Kazienko, P., Kocoń, J., Cichecki, I., Kaszyca, O., & Szydło, D. (2023) · doi:10.36227/techrxiv.170326689.99692093/v1 · Cited by 1
Self-citationGenerative Ai as (Un)Welcome Agents in Medical Crowdfunding: The Trust Dilemma and Moral HazardPreprint
Arnold, C., Xu, L. Z., Saffarizadeh, K., & Madiraju, P. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4725206 · Cited by 1
Exploring the Capabilities of ChatGPT in Women’s HealthPreprint
Bachmann, M., Duta, I., Mazey, E., et al. (2024) · medRxiv · doi:10.1101/2024.02.27.23300005 · Cited by 1
Does Social Bot Help Socialize? Evidence from a Microblogging PlatformPreprint
Gao, Y. A., Zhang, M. M., & Lysyakov, M. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4786302 · Cited by 1
Saif, N., Ali, S., Shaheen, I., Khan, I. U., Khan, S. U., & Khan, M. U. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4855571 · Cited by 3
Kang, S., Odom, W., Chen, A. Y. S., & Neustaedter, C. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790664
Plisiecki, H., Koc, P., Flakus, M., & Pokropek, A. (2025) · Quality & Quantity · doi:10.1007/s11135-025-02116-8 · Cited by 3
Macanovic, A., & Przepiorka, W. (2022) · doi:10.31235/osf.io/cerzs
Transductive Learning for Textual Few-Shot Classification in API-based Embedding ModelsPreprint
Colombo, P., Pellegrain, V., Boudiaf, M., Storchan, V., Tami, M., Ayed, I. B., Hudelot, C., & Piantanida, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.13998
Efficiency Evaluation of ChatGPT for Adverb Type Categorization
Sewunetie, W. T., & Kovács, L. (2023) · World Symposium on Digital Intelligence for Systems and Machines (DISA) · doi:10.1109/disa59116.2023.10308928
Beyond Human Review: Levereging ChatGPT for Label Noise DetectionPreprint
Cichecki, I., Kocoń, J., Kazienko, P., Kaszyca, O., Szydło, D., & Kochanek, M. (2023) · doi:10.36227/techrxiv.170326715.56351742/v1
Self-citationПерспективи та проблеми використання CHATGPT у роботі редактора та журналіста
Фіялка, С. Б. (2023) · Обрії друкарства · doi:10.20535/2522-1078.2023.2(14).295177 · Cited by 1
Social Media vs. Surveys: A New Scalable Approach to Understanding Legislators’ DiscoursePreprint
González‐Rostani, V., Incio, J., & Lezama, G. (2024) · doi:10.31219/osf.io/td3ub
Foppiano, L., Lambard, G., Amagasa, T., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.11052
Uncovering Sentiments Expressed on GPT in Academic Papers
Hong, A., Nwoji, S. C., Murungi, J. K., Tatikonda, P. K., & Kaluvakolli, A. (2023) · International Conference on Open Source Systems and Technologies (ICOSST) · doi:10.1109/icosst60641.2023.10414229
From Silicon to Solutions: AI's Impending Impact on Research and DiscoveryPreprint
Markowitz, D. M., Boyd, R. L., & Blackburn, K. G. (2024) · doi:10.31234/osf.io/dwx5n
Smart contract generation model based on code annotation and AST-LSTM tuningPreprint
Yong, C., Defeng, H., Xu, C., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-3995739/v1
Differentially Private Support Vector Machines with Knowledge Aggregation
Wang, T., Zhang, Y., Liang, J., Wang, S., & Liu, S. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.048115 · Cited by 1
Gs-Cbr-Kbqa: Graph-Structured Case-Based Reasoning for Knowledge Base Question AnsweringPreprint
Li, J., Luo, X., & Lu, G. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4769523
ChatGPT és más nagy nyelvi modellek (LLM-ek) biztonsági kérdései, szervezeti és társadalmi hatásai
Nagy, P. (2024) · Rendvédelem · doi:10.53793/rv.2024.2.1 · Cited by 1
Wang, C., Li, S., Lin, N., et al. (2024) · doi:10.2196/preprints.59435
Liebrenz, M., Smith, A. J., Graña, J., & Buadze, A. (2024) · doi:10.2196/preprints.59109
A Large Language Model Pipeline to Automate the Solution of Competitive Programming Problems
Jayachandran, D., & Blum, J. J. (2024) · ASEE Middle Atlantic Section Spring Conference · doi:10.18260/1-2--45702
ChatGPT Kecerdasan Buatan untuk Inovasi UKM: Tinjauan Sistematis Literatur dan Analisis Bibliometrik
Rozi, F., & Andrian, R. (2024) · Didaktika Jurnal Kependidikan · doi:10.58230/27454312.579
Tutoring and Education: Identifying the Opportunities and Challenges of ChatGPT Among Students
Atiyah, A. N. (2024) · Communications in Mathematics and Applications · doi:10.26713/cma.v14i5.2687
Large Language Models for Data Extraction in Slot-Filling Tasks
Bazan, M., Gniazdowski, T., Wolkiewicz, D., Sarna, J., & Marchwiany, M. E. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-61857-4_1
Powerful AI in Social Science Research
Singh, S., & Aggarwal, R. K. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-3015-9.ch012
Les adjectifs évaluatifs dans le discours juridictionnel : Analyse de corpus
Dolata-Zaród, A. (2024) · Studia Neophilologica · doi:10.1080/00393274.2024.2350713
Target-Phrase Zero-Shot Stance Detection: Where Do We Stand?
Motyka, D., & Piasecki, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63751-3_3 · Cited by 1
Self-citationPotenziare il Giudizio Descrittivo nella Scuola Primaria con l’uso dell’IA generativa
di Padova, M., Tinterri, A., Basta, A., Amatori, G., & Dipace, A. (2024) · IUL Research · doi:10.57568/iulresearch.v5i9.529
Making ChatGPT Work For MePreprint
Keppler, S. M., Sinchaisri, W. P., & Snyder, C. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4700354
ChatGPT for Automated Qualitative Research: Content Analysis (Preprint)Preprint
Bijker, R., Merkouris, S., Dowling, N. A., et al. (2024) · doi:10.2196/preprints.59050
Improving Zero-Shot Stance Detection by Infusing Knowledge from Large Language Models
Guo, M., Jiang, X., & Liao, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5618-6_11 · Cited by 2
Guan, I. Y. N., Zhang, G., Liu, X., Zhao, E., & Wu, J. (2024) · International Conference on Electrical Engineering, Control and Robotics (EECR) · doi:10.1109/eecr60807.2024.10607324
A top-view indoor localization based on discrete distillation of CLIP
Zhao, A., Liu, Y., Cheng, K., Ma, A., & Yu, J. (2024) · Journal of Physics Conference Series · doi:10.1088/1742-6596/2816/1/012018
Automating the Conducting of Surveys Using Large Language Models
Tewari, T., & Hosein, P. A. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-66705-3_9
Yang, H., Xiao, L., Zhu, R., Liu, Z., & Chen, J. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00165
Facts and Tones in Earnings Transcripts *Preprint
Gong, Z., & Li, K. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4950924
InsideOut: Unifying Emotional LLMs to Foster Empathy
Mozikov, M., Severin, N., Glushanina, M., Baklashkin, M., Savchenko, A., & Makarov, I. (2024) · Frontiers in artificial intelligence and applications · doi:10.3233/faia241039
Bernstein, E., Ramsamooj, A., Miller, K. L., & Lum, Z. C. (2024) · doi:10.2196/preprints.68603
Collaborating Digital Twins for Health Coaching
Salman, S., & Richards, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70415-4_20
Kotzé, E., & Senekal, B. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-78255-8_10
Wessling, N., L’Orange, E., Bleckmann, L.-S., et al. (2024) · Springer eBooks · doi:10.1007/978-3-658-45132-5_12
Augmented Reality Research: Benefit or Detriment for Neurodiverse People
Jones, D., Gračanin, D., & Azab, M. (2024) · IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct) · doi:10.1109/ismar-adjunct64951.2024.00015
Using Annotator Labels Instead of Golden Labels for Fine Emotion Detection
Lu, A. L. H., & Iwaihara, M. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0865-2_19 · Cited by 1
Use of large language models to identify fake information
Lande, D., & Hyrda, V. (2024) · Collection Information technology and security · doi:10.20535/2411-1031.2024.12.2.315743
Patil, D. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5057428
Metadata
- Authors
- Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., Kocoń, A., Koptyra, B., Mieleszczenko-Kowszewicz, W., Miłkowski, P., Oleksy, M., Piasecki, M., Radliński, Ł., Wojtasik, K., Woźniak, S., & Kazienko, P.
- Affiliations
- Wrocław University of Science and Technology (PL)
- Corresponding author
- Kocoń, Jan
- ORCID
- Venue
- Information Fusion
- Publisher
- Elsevier BV
- Published
- 2023-11
- Volume
- 99
- Pages
- 101861
- ISSN
- 1566-2535 (ISSN-L)
- OpenAlex
- W4379259169
- Semantic Scholar
- 5848737f78397f72ceae2ba6f3419a6a8502b8ba
- Cited by
- 758
- References
- 105
- Citations by year
- 2026: 95 · 2025: 234 · 2024: 313 · 2023: 115
- FWCI
- 71.55
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Information Fusion (publishedVersion, cc-by)
- SSRN Electronic Journal (acceptedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Ministerstwo Edukacji i Nauki
- Politechnika Wrocławska
- HORIZON EUROPE Framework Programme
- European Regional Development Fund (01-00-0288/22, POIR.01.01, POIR.04.02.00-00C002/19)
- European Commission Marie Sklodowska-Curie Actions (101086321)
- Department of Artificial Intelligence, Korea University
- Narodowe Centrum Nauki (2021/41/B/ST6/04471)
- European Research Executive Agency
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- ChatGPT, GPT-4, large language models, automated assessment, sentiment analysis, emotion recognition, stance detection, word sense disambiguation, linguistic acceptability, question answering, few-shot prompting, algorithmic bias, human-machine interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
