The DOI matches its Crossref record; 80 of the 105 references it cites were found in a registry or an index; none of the 15 sources its claims cite is checked for support yet.
ChatGPT: Jack of all trades, master of none
Open accessAbstract
OpenAI has released the Chat Generative Pre-trained Transformer (ChatGPT) and revolutionized the approach in artificial intelligence to human-model interaction. The first contact with the chatbot reveals its ability to provide detailed and precise answers in various areas. Several publications on ChatGPT evaluation test its effectiveness on well-known natural language processing (NLP) tasks. However, the existing studies are mostly non-automated and tested on a very limited scale. In this work, we examined ChatGPT’s capabilities on 25 diverse analytical NLP tasks, most of them subjective even to humans, such as sentiment analysis, emotion recognition, offensiveness, and stance detection. In contrast, the other tasks require more objective reasoning like word sense disambiguation, linguistic acceptability, and question answering. We also evaluated GPT-4 model on five selected subsets of NLP tasks. We automated ChatGPT and GPT-4 prompting process and analyzed more than 49k responses. Our comparison of its results with available State-of-the-Art (SOTA) solutions showed that the average loss in quality of the ChatGPT model was about 25% for zero-shot and few-shot evaluation. For GPT-4 model, a loss for semantic tasks is significantly lower than for ChatGPT. We showed that the more difficult the task (lower SOTA performance), the higher the ChatGPT loss. It especially refers to pragmatic NLP problems like emotion recognition. We also tested the ability to personalize ChatGPT responses for selected subjective tasks via Random Contextual Few-Shot Personalization, and we obtained significantly better user-based predictions. Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI. Our results provide the basis for a fundamental discussion of whether the high quality of recent predictive NLP models can indicate a tool’s usefulness to society and how the learning and validation procedures for such systems should be established.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper evaluates ChatGPT on 25 analytical NLP tasks and GPT-4 on five subsets, automating prompts and analyzing more than 49k responses. It finds ChatGPT never reaches SOTA performance, with an average loss of about 25%, and that losses are larger for harder, pragmatic, and especially emotion-related tasks.
Key findings
- ChatGPT underperformed SOTA on every task: average SOTA quality was 73.7% versus ChatGPT’s 56.5%, with an average loss of 25.5%; ChatGPT was also less stable (std. dev. 23.3% vs. 16.7%). Excluding eight emotion-related tasks, average loss fell to 12.8% (ChatGPT 69.7% vs. SOTA 80.0%).
- Loss correlated with task difficulty (Pearson 0.46); pragmatic tasks were harder on average (32.3% difficulty) than semantic tasks (15.6%), and emotion tasks had an average loss of 52.59% (ChatGPT 28.44% vs. SOTA 60.28%).
- Random Contextual Few-Shot Personalization improved user-based predictions: GoEmoPer3 was 4.08 percentage points better than GoEmoPer0, while AggressionPer and UnhealthyPer showed higher ChatGPT loss than their non-personalized counterparts by 3.25 and 12.55 percentage points, respectively.
- GPT-4 had small loss on semantic tasks (0.9%–6.4% for TextEntail and WNLI) but still high loss on emotional tasks (46.3%–56.3%); ChatGPT still outperformed GPT-4 on GoEmo and PolEmo. Qualitative analysis found bias, moralizing refusals, non-determinism, and cases where experts accepted ChatGPT answers over human labels (e.g., Unhealthy: 81% ChatGPT vs. 43% human).
Methods
- Tested 25 NLP datasets/tasks (23 English, 2 Polish), split into semantic and pragmatic categories, including sentiment, emotion, offensiveness, stance, WSD, NLI, and QA; some tasks used personalized user context.
- Automated prompting via an unofficial PyGPT API using up to 20 accounts, with manual post-processing; the paper reports over 38,000 prompts in the setup and more than 49k responses in the abstract.
- Compared ChatGPT to SOTA models by rerunning available source code or using reported metrics, using F1 Macro or accuracy and computing Loss = 100% × (SOTA − ChatGPT) / SOTA; GPT-4 was tested on five selected subsets.
Limitations
- Stated: prompts may not be strict or precise enough; post-processing was sometimes required; ChatGPT produced unexpected out-of-range responses; internal policy and biases affected answers; context was generally respected but offered limited control; efficiency and costs were constrained by lack of an official API, with about 2–3k prompts per day achieved.
- Stated: the difficulty analysis is limited because only 25 tasks were considered; GPT-4 testing was restricted by access limits (25 prompts per 3 hours).
- Evident from the text: dataset availability for ChatGPT training was assessed manually and is uncertain; qualitative expert evaluation covered 100 randomly selected cases per dataset and involved subjective judgments; ChatGPT showed non-determinism, with results varying across launches.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
98 Own result · 37 Own interpretation · 34 No basis given · 11 Own method · 10 Not checked yet
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For GPT-4 model, a loss for semantic tasks is significantly lower than for ChatGPT.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We showed that the more difficult the task (lower SOTA performance), the higher the ChatGPT loss.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Simultaneously, ChatGPT was less stable: the standard deviation of its performance was 23.3% compared to only 16.7% for the SOTA solutions.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“because for all of these datasets, we observe a significant drop in quality relative to state-of-the-art models.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It especially refers to pragmatic NLP problems like emotion recognition.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also tested the ability to personalize ChatGPT responses for selected subjective tasks via Random Contextual Few-Shot Personalization, and we obtained significantly better user-based predictions.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It is usually greater for semantic tasks rather than for pragmatic ones, which is related to the task difficulty, see Sec. 6.2.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our comparison of its results with available State-of-the-Art (SOTA) solutions showed that the average loss in quality of the ChatGPT model was about 25% for zero-shot and few-shot evaluation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“yet it has them in topics regarding ideological views.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 46
- Probably not a substantial claim. 70
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The crucial finding from our studies is that the ChatGPT performance is always lower than the SOTAmethods (loss>0) in all the tasks considered.”
- Also stated in: § 6.1. Jack of all trades, master of none · p. 10 · plain
“It means that ChatGPT never reached the level of the best existing models.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 6.1. Jack of all trades, master of none · p. 10 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The loss for most tasks did not exceed 25%. It was greater only for three problems: GoEmotions, PolEmo, and TweetEmoji.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The average quality of SOTA methods was at 73.7%, whereas ChatGPT was at only 56.5%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while GPT-4 achieved significantly higher scores in the remaining three semantic tasks with even a 9.9 p.p. increase for the WNLI task.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While solving mathematical calculations (SQuAD), the highest loss was among semantic tasks: 23.7%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In general, pragmatic tasks are more difficult (average difficulty = 32.3%), while the average difficulty for semantic tasks is only 15.6%.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also noticed that when it is unable to perform a task on the provided example, it will refuse to do so and provide an explanation why”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Those situations are usually solved by supplementing the model with information.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It comes especially from the emotional tasks, which are pragmatic and very difficult (average 39.7%).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the case of mathematical reasoning, it can provide a whole explanation of how it reached a certain outcome and provide only the answer without explanation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When exploiting the possibilities of ChatGPT, we can see that it can perform various tasks, including recognizing generalized and personalized dimensions of Natural Language Processing, answering questions where a generous amount of domain knowledge is required, or even writing lines of code in the programming language of choice.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We can also observe that the loss is correlated with the task difficulty; see Fig. 7. The Pearson correlation coefficient between difficulty and loss is equal to 0.46.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When conducting experiments regarding tasks such as humor recognition or offensiveness detection, we have stumbled upon output that not only refuses to answer whether something is or is not funny but also sends a moralizing message with an irritated tone.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ChatGPT can also return a number as a category indicator or whole output in the JSON format.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In such a case, the average loss is reduced by as much as half, from 25.5% to 12.8%; the difference in performance drops from 17.2pp to 10.3pp.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If we discard all eight emotion-related tasks (ids: 12-17, 20-21), the average SOTA performance reaches 80%(increase by 6.3pp), but ChatGPT improves much more: by 13.2pp, up to 69.7%.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ChatGPT will only generate the content if the user adequately motivates it with the scientific goals.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, in the case of the ChatGPT model, prompt tuning is technically unavailable”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“as it has happened in the case of ClarinEmo B.12, where the model stated that all provided texts are legal and financial statements.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Anyway, the differences in performance are not significant.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Onthe other hand, ChatGPT demonstrated its instability and tendency towards non-determinism.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It also refers to the necessary manual correction of typos surprisingly provided by ChatGPT”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, prompts themselves and the sequence of prompts provide limited control over them.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“So it evaluates the main content but ignores the broader context (see Chat 64).”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Therefore, some responses are locked due to the OpenAI policy.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“e.g., the unlisted emotion satisfaction has been delivered:”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In this example, the sentence was interpreted as not aligned with society's standards.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“## Who is right? (expert rate)”
- Also stated in: § Expected · p. 41 · plain
“Evaluation result human - not correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Also stated in: § Expected · p. 41 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“## Who is right? (expert rate)”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluation result human - not correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“## Who is right? (expert rate)”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“human - correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluation result human - not correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The annotator has evaluated the below message as expressing gratitude, whereas ChatGPT regards it as neutral (Chat 58)”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“3. ChatGPT flattens the message, partially ignoring the metatext.”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“At the same time, ChatGPT explicitly distances itself from judging people.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ChatGPT labeled as aggressive only 11 texts from the WikiDetox Aggression dataset labeled by people as non-aggressive, while the opposite decision was taken 207 times.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Evaluation result human - not correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Restrictive or wrongly implemented policy will influence the user experience of the ChatGPT by not answering all user's prompts exaggeratedly judging many questions not along with OpenAI rules:”
hedgeddirect- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“human - correct, ChatGP - not correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“when asked which language phenomena demonstrate the enthusiasm or positive sentiment in the text, it gives a list containing such phenomena as the use of superlatives or lack of negative language”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Context is generally respected by ChatGPT.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“human - correct, ChatGP - correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The problem of controversial and ambiguous utterances is solved by ChatGPT's demonstration of a lower level of confidence”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“## Who is right? (expert rate)”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“human - not correct , ChatGPT - not correct”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“## Who is right? (expert rate)”
plainSingle case- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As mentioned in the previous section, ChatGPT tends to find negative connotations in the given text.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63:”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Similarly, ChatGPT justifies the negative sentiment of the news about the ban on naming streets after Fidel Castro, Chat 66:”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Internal policy can strongly influence the answers received.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All these tasks are related to a very subjective problem of emotional perception and individual interpretation of the content.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“In cases where the performance of ChatGPT was compared to other solutions (e.g. [26, 30, 41]), it worked on a comparable level to the competitor but not outperforming any major SOTA solutions.”
What the sources say
- Not checked yetcites [26] Kung, 2022
- Not checked yetcites [30] Goyal, 2022
- Not checked yetcites [41] Jiao, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“There were a couple of large-scale evaluations published recently [46, 47] and both of them indicate that while the performance of ChatGPT is significant it doesn't outperform the SOTA solutions with the exception of sentiment analysis task in case of [46] which suggests it is a significant area for further research.”
What the sources say
- Not checked yetcites [46] Amin, 2023
- Not checked yetcites [47] Bang, 2023
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
- Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
In the paper
“ChatGPT's successor, the GPT-4 [19], is most likely an even larger model that can additionally receive not only text but also images as input.”
What the sources say
- Not checked yetcites [19] OpenAI, 2023
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“Language models are prone to generating responses containing human-like biases as well as presenting moral and ethical stances [23].”
What the sources say
- Not checked yetcites [23] Schramowski, 2022, Large pre-trained language models contain human-like biases of what is right and wrong to do, Nat. Mach. Intell. source
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even though the creators of ChatGPT secure the model against answers that are offensive, there are still multiple methods that may reveal its hidden biases.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nevertheless, we can state that ChatGPT performs pretty well on all tasks except emotional ones.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, the number of cases that required post-processing was relatively small (column #Post-processed in Tab. 1). For most tasks (16), the contribution of such texts was less than 5%. Only for Aggression, SQuAD, Unhealthy, and ReAding, it exceeded 15%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Manual post-processing correctly extracted and mapped ChatGPT's raw outputs to the intended labels.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Simultaneously, its achievements are always below SOTA but usually not so much.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such results prove that ChatGPT is Jack of all trades, master of none .”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the other hand, using the prompts directly as humans designed them implicitly allows us to evaluate models' language comprehension abilities.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We emphasize that the results for GPT-4, as for ChatGPT, could significantly differ, if distinct prompt schemes are compared, see Sec. 6.6.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“If possible, we launched our models equivalent to SOTA solutions since the setup (especially data split) was often different than in the original paper. For that purpose, we usually utilized source codes published by the authors. Unfortunately, it was impossible for some tasks, so we exploited the performance results provided in the original paper.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Rests on an assumption: The SOTA results taken from original papers are comparable to the paper's own setup despite differences in data split and evaluation.
In the paper
“Borji [105] conducted a systematic review of the typical categories of ChatGPT failures.”
What the sources say
- Not checked yetcites [105] Korczyński, 2022, Compression methods for transformers in multidomain sentiment analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The above errors are derived from both incorrect reasoning in terms of psychological reasoning and bias and discrimination.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This analysis, however, requires further investigations since the number of the tasks considered (25) still remains relatively small.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It is observable that semantic tasks (blue crosses) are rather easy; hence, their ChatGPT loss is relatively small.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The annotations of six independent annotators were aggregated to label each sentence with all potential options, using the label when at least two annotators agreed on it.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Using the glosses to explain senses to a language model implicitly tests its language comprehension abilities.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“It is our new dataset that has not yet been published. We exploited this dataset to ensure that ChatGPT was not trained on it.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: ChatGPT was not trained on the ClarinEmo dataset.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and the only way to verify prompt relevance is to evaluate its performance directly in the downstream task.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“If available, we tried to validate ChatGPT using one measure F1 Macro, which is commonly acceptable for imbalanced data, Tab. 2.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In the case of CoLa, WNLI, WordContext, and MathQA, we had to rely on the accuracy, as it was the only one presented in the reference paper; we could not replicate their studies and calculate our measures. WNLI and WordContext have their two classes balanced, so it is not an issue.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“which further implies possible consequences regarding clashes with fake news.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It may suggest that providing some answerable questions helps it detect unanswerable ones with the same context.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Demonstration-based personalization included in our prompts can be treated as similar to few-shot learning, even an extended context to a given prompt.”
plaincorrelationaln = 300- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Hagendorff [104] drew attention to the fact that chatbot ethics can be a subject of debate in fairness, nondiscrimination, and justice.”
What the sources say
- Not checked yetcites [104] Borji, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result, the raw text provided by ChatGPT may require extraction of crucial information, e.g., final labels.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Information about the distinction for a particular footballer is neutral.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“1. Prompts may not be strict and precise enough.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“2. ChatGPT seems to have no regard for individuals, instead judging situations.”
hedgedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, this often leads to mistakes”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“6. ChatGPT presents the sense of common human morality.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“ChatGPTrequires prompts in natural language, which is - by nature - not structured and can be different for different users and tasks.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“A small difference in these two values proves a similar distribution of classes in both sets, thus, a good stratification of sampling.”
- Also stated in: § Table 8 · p. 28 · boosted
“A marginal difference in entropy of output real values for test or dev set (column #Test in Tab. 1) and within the set used by us (column #Used in Tab. 1) demonstrates a good stratification of the used set selection.”
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Table 8 · p. 28 · boosted
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It also depends on the prompting quality and correctness.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its sentiment, however, can be both positive and negative. It depends on the sympathies of the recipient sympathies regarding specific footballers.”
plainconditionalSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT evaluates rather situations than participants, but words refer to people, which lead to a specific, predetermined assessment”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Analysis was conducted by experts who are specialists trained in the recognition of emotions in the text. One of them is a psychologist and another is a linguist, both are experienced annotators.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It also provides a unique selfexplanation capability that facilitates human understanding and adaptation to the expected outcome.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Thus, we are dealing with self-explanatory artificial intelligence, which is a part of eXplainable Artificial Intelligence (XAI)”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It includes various biases and political correctness incorporated into the general ChatGPT model.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper relies on this without support.
The 25 selected NLP tasks are representative of typical NLP analytical tasks.
“We tested ChatGPT on 25 tasks focusing on solving common NLP problems and requiring analytical reasoning, Tab. 1.”
The paper relies on this without support.
ChatGPT's self-reports about whether it used a dataset for training are reliable.
“Additionally, we asked ChatGPT whether or not the dataset was used for training.”
The paper relies on this without support.
ChatGPT was not trained on the ClarinEmo dataset.
“It is our new dataset that has not yet been published. We exploited this dataset to ensure that ChatGPT was not trained on it.”
The paper relies on this without support.
The unofficial PyGPT API and the use of up to 20 accounts produced outputs equivalent to the official ChatGPT service.
“Without an official API, we modified and used an unofficial API called PyGPT 5 , written in Python. During the research, we exploited up to 20 accounts to gather data regarding 25 datasets.”
The paper relies on this without support.
The SOTA results taken from original papers are comparable to the paper's own setup despite differences in data split and evaluation.
“If possible, we launched our models equivalent to SOTA solutions since the setup (especially data split) was often different than in the original paper. For that purpose, we usually utilized source codes published by the authors. Unfortunately, it was impossible for some tasks, so we exploited the performance results provided in the original paper.”
The paper relies on this without support.
Manual post-processing correctly extracted and mapped ChatGPT's raw outputs to the intended labels.
“This imposes a necessity to check what happened and extract answers from ChatGPT output manually. The next step is to cast the resulting outputs to the correct labels in the dataset.”
The paper relies on this without support.
The performance of modern language models such as T5, GPT-3, and ChatGPT heavily relies on the quality of task-specific prompts.
“The performance of modern language models, such as T5, GPT-3, and ChatGPT, heavily relies on the quality of task-specific prompts.”
The paper relies on this without support.
The prompt-based learning paradigm requires careful prompt engineering and prompt tuning.
“The prompt-based learning paradigm requires careful prompt engineering and prompt tuning.”
The paper relies on this without support.
Most tasks require a prompt that enables the model to choose a certain value from the provided options.
“Most tasks require a prompt that enables the model to choose a certain value from the provided options.”
The paper relies on this without support.
The annotations in the selected 4 datasets were of a more or less subjective nature, and detailed guidelines were not necessary to achieve high interannotator agreement.
“The annotations in the selected 4 datasets were of a more or less subjective nature, and for this reason, it was not necessary to create detailed guidelines in order to achieve high interannotator agreement”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In a large study of ChatGPT that included 25 tasks and over 48,000 prompts, the tool was found to perform reasonably well when conducting quantitative analysis, with a somewhat poorer performance with tasks that require more reasoning ( Kocoń et al., 2023).”
From Generative AI for corpus approaches to discourse studies: A critical evaluation of ChatGPT (2024) · cites this paper as Kocoń et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This underperformance by the chat model is consistent with the findings by Kocon et al. [39].”
From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [39]
In the citing paper
“Similarly, LLMs’ performance in logic and deductive tasks is often poor (64, 65), so using them for analysis may lead to false conclusions.”
From How should the advancement of large language models affect the practice of science? (2025) · cites this paper as [64]
In the citing paper
“Kocoń et al. (76) found that LLMs may struggle with complex, subjective tasks such as emotion recognition, and Huang et al. (18) found that LLMs can reinforce lay perspectives when detecting implicit hate speech and may thus mislead nonexperts with their explanations in cases where they make incorrect decisions.”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [76]
In the citing paper
“However, this perspective of LLMs, as a convenient and comprehensive tool in psychological text analysis, is challenged by recent critiques emphasizing their limitations, such as inconsistencies in text annotations, difficulties in explaining complex constructs such as implicit hate speech, and a potential lack of depth in specialized or sensitive areas (18, 76, 77).”
From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [76]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| 2 | AggressionPer | Pragmatic | F1 Macro | Our | 72.57 | 81.03 | 8.46 | 19.97 | 10.44 |”
Reported 19.97 · recomputed 18.97
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 6.3. Random Contextual Few-Shot Personalization · p. 12
“| 24 | ReAding | Semantic | F1 Macro | Our | 76.36 | 84.71 | 8.35 | 15.29 | 9.86 |”
§ Prompt · p. 29
“| 24 | ReAding | 76.36 | 76.34 | - | - | 84.71 | - |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 6.3. Random Contextual Few-Shot Personalization · p. 12
“| 8 | TextEntail | Semantic | F1 Macro | Paper | 88.09 | 92.10 | 4.01 | 7.90 | 4.35 |”
§ Prompt · p. 29
“| 8 | TextEntail | 88.09 | 87.88 | - | - | 92.1 | - |”
Citations
Cites
1 retracted
RETRACTED: WELMSD – word embedding and language model based sarcasm detection
Kumar, P., & Sarin, G. (2022) · Online Information Review · doi:10.1108/oir-03-2021-0184 · Cited by 17 · cited 1×
“3%) 995 2 20137 /19643 5 Sarcasm P EN Humor recognition No Binary classification Sarcasmania [67] / [68] 3 Yes 5967 1000 10 61 (6.”
BackgroundRetractedBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kilgarriff, A. (2000) · Language · doi:10.2307/417141 · Cited by 11,529
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
A survey of deep neural network architectures and their applications
Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y., & Alsaadi, F. E. (2017) · Neurocomputing · doi:10.1016/j.neucom.2016.12.038 · Cited by 3,239
Research works printed but not linked: 6
- Johnson, 2016, Supervised and semi-supervised text categorization using LSTM for region embeddings, vol. 48, 526
- Gillioz, 2020, Overview of the transformer-based models for NLP tasks, vol. 21, 179
- Radford, 2018, Improving language understanding by generative pre-training, OpenAI
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI
- Ngo, 2022, StudEmo: A non-aggregated review dataset for personalized emotion recognition, 46
- Fellbaum, 1998, Wordnet: An electronic lexical database, Comput. Linguist., 292
Cited by
Sorted by citations
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855
“In terms of stability, reference [43] concludes that ChatGPT’s performance is always lower than SOTA, the current state-of-the-art model, in almost all tasks.”
BackgroundUnlocking the opportunities through ChatGPT Tool towards ameliorating the education system
Javaid, M., Haleem, A., Singh, R. P., Khan, S., & Khan, I. H. (2023) · BenchCouncil Transactions on Benchmarks, Standards and Evaluations · doi:10.1016/j.tbench.2023.100115 · Cited by 535
Yu, H. (2023) · Frontiers in Psychology · doi:10.3389/fpsyg.2023.1181712 · Cited by 409
“Only then can we ensure that Chat GPT has a positive impact on education and achieves long-term sustainability (Kocoń et al., 2023).”
BackgroundRWKV: Reinventing RNNs for the Transformer Era
Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Derczynski, L., et al. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.936 · Cited by 390
Self-citationBang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Data
Xu, X., Yao, B., Dong, Y., Gabriel, S., Yu, H., Hendler, J., Ghassemi, M., Dey, A. K., & Wang, D. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3643540 · Cited by 278
“Some work explored the capability of LLMs for sentiment analysis and emotion reasoning [56, 81, 115].”
BackgroundGenghis Khan shark optimizer: A novel nature-inspired algorithm for engineering optimization
Hu, G. Y., Guo, Y., Wei, G., & Abualigah, L. (2023) · Advanced Engineering Informatics · doi:10.1016/j.aei.2023.102210 · Cited by 272
A comprehensive survey of ChatGPT: Advancements, applications, prospects, and challenges
Nazir, A., & Wang, Z. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100022 · Cited by 245
Younis, H. A., Eisa, T. A. E., Nasser, M., Sahib, T. M., Noor, A. A., Alyasiri, O. M., Salisu, S., Hayder, I. M., & Younis, H. A. (2024) · Diagnostics · doi:10.3390/diagnostics14010109 · Cited by 242
Is chatGPT a good nlg evaluator? A preliminary study
Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J. (2023) · Proceedings of the 4th New Frontiers in Summarization Workshop · doi:10.18653/v1/2023.newsum-1.1 · Cited by 235
Artificial muses: Generative artificial intelligence chatbots have risen to human-level creativity
Haase, J., & Hanel, P. H. P. (2023) · Journal of Creativity · doi:10.1016/j.yjoc.2023.100066 · Cited by 233
Bad Actor, Good Advisor: Exploring the Role of Large Language Models in Fake News Detection
Hu, B., Sheng, Q., Cao, J., Shi, Y., Li, Y., Wang, D., & Qi, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i20.30214 · Cited by 288
“Results show that LLMs may not have comprehensive superiority compared with a well-trained small model in some types of NLU tasks (Zhong et al., 2023; Koco´n et al., 2023).”
MethodLai, V., Ngo, N., Pouran Ben Veyseh, A., Man, H., Dernoncourt, F., Bui, T., & Nguyen, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.878 · Cited by 233
Albayati, H. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100203 · Cited by 188
Kofinas, A. K., Tsay, C. H., & Pike, D. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13585 · Cited by 207
Factuality challenges in the era of large language models and opportunities for fact-checking
Augenstein, I., Baldwin, T., Cha, M., Chakraborty, T., Ciampaglia, G. L., Corney, D., DiResta, R., Ferrara, E., Hale, S., Halevy, A., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00881-z · Cited by 187
“Recent studies suggested that ChatGPT is more inclined to excel in stance detection tasks than in tasks involving emotions or pragmatic analysis, even though it might not have undergone pretraining on evaluation sets explicitly tailored to stance detection tasks 106 .”
AI–Human Hybrids for Marketing Research: Leveraging Large Language Models (LLMs) as Collaborators
Arora, N., Chakraborty, I., & Nishimura, Y. (2024) · Journal of Marketing · doi:10.1177/00222429241276529 · Cited by 211
“Some articles fi nd that although LLMs outperform expert humans for certain less ambiguous labeling tasks, their performance is subpar for tasks that require superior human judgement (Kocon ́ et al. 2023; Ziems et al. 2024).”
Kabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642596 · Cited by 160
Towards Interpretable Mental Health Analysis with Large Language Models
Yang, K., Ji, S., Zhang, T., Xie, Q., Kuang, Z., & Ananiadou, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.370 · Cited by 155
“There are also works on analyzing the emotional reasoning ability of ChatGPT including (Qin et al., 2023; Zhong et al., 2023; Kocoń et al., 2023; Chen et al., 2023) on the sentiment classification task, where ChatGPT achieves comparable or worse performance compared with fine-tuning based methods based on PLMs.”
BackgroundResultInfluentialChatGPT for Automated Qualitative Research: Content Analysis
Bijker, R., Merkouris, S., Dowling, N. A., & Rodda, S. N. (2024) · Journal of Medical Internet Research · doi:10.2196/59050 · Cited by 224
“As described previously [49], ChatGPT-generated data commonly require various postprocessing steps to clean the data and check whether responses are in line with the instructions.”
Hao, X., Demir, E., & Eyers, D. R. (2024) · Technology in Society · doi:10.1016/j.techsoc.2024.102662 · Cited by 205
A Culturally Sensitive Test to Evaluate Nuanced GPT Hallucination
McIntosh, T. R., Liu, T., Susnjak, T., Watters, P., Ng, A., & Halgamuge, M. N. (2024) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3332837 · Cited by 145
“Despite their usefulness, existing classifications often fall short due to the subjectivity of language interpretation [28], [29].”
BackgroundAkpan, I. J., Kobara, Y. M., Owolabi, J., Akpan, A. A., & Offodile, O. F. (2024) · International Transactions in Operational Research · doi:10.1111/itor.13522 · Cited by 187
Google DeepMind’s gemini AI versus ChatGPT: a comparative analysis in ophthalmology
Masalkhi, M., Ong, J., Waisberg, E., & Lee, A. G. (2024) · Eye · doi:10.1038/s41433-024-02958-w · Cited by 130
“Large language models such as ChatGPT can make errors in understanding the context of information, or provide outdated information, which further complicates the usage of these technologies in a clinical context [8–11].”
MethodRaghu Raman, R., Mandal, S., Das, P., Kaur, T., Sanjanasri, J. P., & Nedungadi, P. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/3085910 · Cited by 288
“For example, engineering and management students will likely perceive ChatGPT di ff erently based on their selective and speci fi c usage [12].”
BackgroundChatGPT is a Remarkable Tool—For Experts
Azaria, A., Azoulay, R., & Reches, S. (2023) · Data Intelligence · doi:10.1162/dint_a_00235 · Cited by 184
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01852 · Cited by 1,172
“In terms of stability, reference [43] concludes that ChatGPT’s performance is always lower than SOTA, the current state-of-the-art model, in almost all tasks.”
BackgroundWachinger, J., Bärnighausen, K., Schäfer, L. N., Scott, K., & McMahon, S. A. (2024) · Qualitative Health Research · doi:10.1177/10497323241244669 · Cited by 125
Large language models: a survey of their development, capabilities, and applications
Yadagiri, A., & Pakray, P. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02310-4 · Cited by 155
Large Language Model‐Based Chatbots in Higher Education
Yigci, D., Eryilmaz, M., Yetisen, A. K., Tasoglu, S., & Ozcan, A. (2025) · Advanced Intelligent Systems · doi:10.1002/aisy.202400429 · Cited by 113
Perils and opportunities in using large language models in psychological research
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae245 · Cited by 111
ChatGPT: perspectives from human–computer interaction and psychology
Liu, J. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1418869 · Cited by 99
Exploring ChatGPT Capabilities and Limitations: A Survey
Koubâa, A., Boulila, W., Ghouti, L., Alzahem, A., & Latif, S. (2023) · IEEE Access · doi:10.1109/access.2023.3326474 · Cited by 188
“Kocon et al. [46] evaluated the capabilities of the ChatGPT on 25 diverse analytical NLP tasks, most of which are subjective in nature, including sentiment analysis, emotion recognition, offensiveness and stance detection, natural language inference, word sense disambiguation, linguistic…”
BackgroundGenerative AI for corpus approaches to discourse studies: A critical evaluation of ChatGPT
Curry, N., Baker, P., & Brookes, G. (2024) · Applied Corpus Linguistics · doi:10.1016/j.acorp.2023.100082 · Cited by 102
Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)
Ahmed, T., Pai, K. S., Devanbu, P., & Barr, E. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639183 · Cited by 112
“This underperformance by the chat model is consistent with the findings by Kocon et al. [39].”
ResultExploring ChatGPT literacy in language education: A global perspective and comprehensive approach
Ma, Q., Crosthwaite, P. R., Sun, D., & Zou, D. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100278 · Cited by 149
Gemini versus ChatGPT: applications, performance, architecture, capabilities, and implementation
Rane, N., Choudhary, S., & Rane, J. (2024) · Journal of Applied Artificial Intelligence · doi:10.48185/jaai.v5i1.1052 · Cited by 172
Physics-informed machine learning in geotechnical engineering: a direction paper
Yuan, B., Choo, C. S., Yeo, L. Y., Wang, Y., Yang, Z., Guan, Q., Suryasentana, S. K., Choo, J., Shen, H. R., Megía, M., et al. (2025) · Geomechanics and Geoengineering · doi:10.1080/17486025.2025.2502029 · Cited by 107
Navigating the Power of Artificial Intelligence in Risk Management: A Comparative Analysis
Yazdi, M., Zarei, E., Adumene, S., & Beheshti, A. (2024) · Safety · doi:10.3390/safety10020042 · Cited by 117
“The existing literature, as referenced by [25], critiques that AIcs often appear to be “ jack of all trades, master of none ”.”
BackgroundYu, S., Androsov, A., Yan, H., & Chen, Y. (2024) · Computers & Education · doi:10.1016/j.compedu.2024.105111 · Cited by 90
Lozić, E., & Štular, B. (2023) · Future Internet · doi:10.3390/fi15100336 · Cited by 115
Vman: visual-modified attention network for multimodal paradigms
Song, X., Han, D., Chen, C., Shen, X., & Wu, H. (2024) · The Visual Computer · doi:10.1007/s00371-024-03563-4 · Cited by 75
Veselovsky, V., Ribeiro, M. H., & West, R. (2023) · arXiv · doi:10.48550/arxiv.2306.07899 · Cited by 73
Can we trust the evaluation on ChatGPT?
Aiyappa, R., An, J., Kwak, H., & Ahn, Y.-Y. (2023) · Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023) · doi:10.18653/v1/2023.trustnlp-1.5 · Cited by 72
Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs
Balloccu, S., Schmidtová, P., Lango, M., & Dusek, O. (2024) · Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.eacl-long.5 · Cited by 70
Uncovering the semantics of concepts using GPT-4
Mens, G. L., Kovács, B., Hannan, M. T., & Pros, G. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2309350120 · Cited by 83
How should the advancement of large language models affect the practice of science?
Binz, M., Alaniz, S., Roskies, A., Aczel, B., Bergstrom, C. T., Allen, C., Schad, D., Wulff, D., West, J. D., Zhang, Q., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401227121 · Cited by 74
Can ChatGPT evaluate research quality?
Thelwall, M. (2024) · Journal of Data and Information Science · doi:10.2478/jdis-2024-0013 · Cited by 68
“They are least accurate for tasks involving understanding and for practical tasks ( Kocoń et al., 2023).”
A Wide Evaluation of ChatGPT on Affective Computing Tasks
Amin, M. M., Mao, R., Cambria, E., & Schuller, B. W. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3419593 · Cited by 92
“Theworksof[41],[42],[43]explorethecapabilitiesofChatGPT onawiderangeofNLPproblemsincludingsentimentanalysis and emotion recognition, and other NLP tasks like NER and text summarising.”
ChatGPT: A brief narrative review
Gupta, B., Mufti, T., Sohail, S. S., & Madsen, D. Ø. (2023) · Cogent Business & Management · doi:10.1080/23311975.2023.2275851 · Cited by 95
“It has the ability to detect typical grammatical constructions and idioms after being trained on a vast corpus of textual data, which includes books, papers, and websites (Kocoń et al., 2023).”
BackgroundFNSPID: A Comprehensive Financial News Dataset in Time Series
Dong, Z., Xinyu, F., & Peng, Z. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3671629 · Cited by 61
“To integrate sentiment labels into the input without intensive human labeling, we utilized ChatGPT for sentiment analysis, ac-knowledging the challenges faced by conventional algorithms and language models, including GPT-2 and GPT-3, in accurately scoring sentiments [12, 16, 24, 50].”
MethodA Review of Research on Spectrum Sensing Based on Deep Learning
Zhang, Y., & Luo, Z. (2023) · Electronics · doi:10.3390/electronics12214514 · Cited by 59
Wang, C., Li, S., Lin, N., Zhang, X., Han, Y., Wang, X., Liu, D., Tan, X., Pu, D., Li, K., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/59435 · Cited by 58
“A critical factor influencing ChatGPT’s response is the prompt input given to the model [28,29].”
Gender biases within Artificial Intelligence and ChatGPT: Evidence, Sources of Biases and Solutions
Ho, J. Q. H., Hartanto, A., Koh, A., & Majeed, N. M. (2025) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2025.100145 · Cited by 83
Automatically Inspecting Thousands of Static Bug Warnings with Large Language Model: How Far Are We?
Wen, C., Cai, Y., Zhang, B., Su, J., Xu, Z., Liu, D., Qin, S., Ming, Z., & Tian, C. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3653718 · Cited by 66
“For comparison, We also include two versions of Llm4sa, one with few-shot prompting (referred to as Llm4sa F ) and another with zero-shot prompting (referred to as Llm4sa Z ).”
MethodCambria, E., Zhang, X., Mao, R., Chen, M., & Kwok, K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-76827-9_11 · Cited by 62
“As also demonstrated by other recent works [24, 42], ChatGPT turned out to be jack of all trades but master of none [28] also in the context of affective computing: while the performance of ChatGPT is acceptable on many different NLP tasks, specialized models like SenticNet 8 (and even RoBERTa in…”
BackgroundHeidari, A., Navimipour, N. J., Zeadally, S., & Chamola, V. (2024) · Internet Technology Letters · doi:10.1002/itl2.530 · Cited by 56
Xie, Z., Wu, X., & Xie, Y. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13032 · Cited by 70
Monib, W. K., Qazi, A., Apong, R. A. A. H. M., Azizan, M. T., Silva, L. D., & Yassin, H. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2105 · Cited by 119
Liu, C., Arulappan, A., Naha, R. K., Mahanti, A., Kamruzzaman, J., & Ra, I.-H. (2024) · IEEE Access · doi:10.1109/access.2024.3445413 · Cited by 66
“The study stands out for its comprehensive examination of various LLMs, including BERT, FinBERT, and ChatGPT, within the specific context of financial sentiment analyses [12], [14], [15].”
BackgroundMethodGöktaş, P. (2024) · Journal of Decision System · doi:10.1080/12460125.2024.2410042 · Cited by 69
HARGPT: Are LLMs Zero-Shot Human Activity Recognizers?
Ji, S., Zheng, X., & Wu, C. (2024) · 2024 IEEE International Workshop on Foundation Models for Cyber-Physical Systems & Internet of Things (FMSys) · doi:10.1109/fmsys62467.2024.00011 · Cited by 111
“1, the raw IoT sensor data is fed directly into various well-known LLMs such as ChatGPT [8], Google Gemini [16], and LLaMA2-70b [17], yielding recognition outcomes.”
MethodKim, M., & Lu, X. (2024) · Journal of English for Academic Purposes · doi:10.1016/j.jeap.2024.101422 · Cited by 61
Chu, C. P. (2024) · Frontiers in Veterinary Science · doi:10.3389/fvets.2024.1395934 · Cited by 72
Evaluation of LLM-based chatbots for OSINT-based Cyber Threat Awareness
Shafee, S., Bessani, A., & Ferreira, P. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125509 · Cited by 101
Exploring ChatGPT Capabilities and Limitations: A Critical Review of the NLP Game ChangerPreprint
Koubâa, A., Boulila, W., Ghouti, L., Alzahem, A., & Latif, S. (2023) · Preprints.org · doi:10.20944/preprints202303.0438.v1 · Cited by 52
Exploring Capabilities of Large Language Models such as ChatGPT in Radiation Oncology
Dennstädt, F., Hastings, J., Putora, P. M., Vu, E., Fischer, G. F., Süveg, K., Glatzer, M., Riggenbach, E., Hà, H.-L., & Čihorić, N. (2023) · Advances in Radiation Oncology · doi:10.1016/j.adro.2023.101400 · Cited by 56
Towards Understanding the Geospatial Skills of ChatGPT
Mooney, P., Cui, W., Guan, B., & Juhász, L. (2023) · ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery (GeoAI) · doi:10.1145/3615886.3627745 · Cited by 73
“While more focussed on the impact of prompting in question asking for ChatGPT, Kocoń et al. [29] state that “it is still an open question what would happen if ChatGPT was finetuned using the datasets for specific tasks”.”
ChatGPT-enabled two-stage auctions for electric vehicle battery recycling
Feng, J., Ning, Y., Wang, Z., Guo, L., & Xu, S. X. (2024) · Transportation Research Part E Logistics and Transportation Review · doi:10.1016/j.tre.2024.103453 · Cited by 52
From Classification to Clinical Insights
Englhardt, Z., Ma, C., Morris, M. E., Chang, C.-C., Xu, X., Qin, L., McDuff, D., Liu, X., Patel, S., & Iyer, V. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3659604 · Cited by 86
“In the mental health domain, LLMs have been explored for applications such as sentiment analysis and emotional reasoning [27, 41, 66].”
MethodHow understanding the limitations and risks of using ChatGPT can contribute to willingness to use
Murshidi, G. H. A., Shulgina, G., Kapuza, A., & Costley, J. (2024) · Smart Learning Environments · doi:10.1186/s40561-024-00322-9 · Cited by 62
“The emergence of ChatGPT in November 2022 as a free and open access GAI tool has attracted significant interest and led to its adoption in various sectors, including education (Dwivedi et al., 2023; Kocoń et al., 2023; Liu et al., 2023a, 2023b; Lund & Wang, 2023).”
BackgroundEvolution of Learning: Assessing the Transformative Impact of Generative AI on Higher Education
Krause, S., Panchal, B. H., & Ubhe, N. (2025) · Frontiers of digital education. · doi:10.1007/s44366-025-0058-7 · Cited by 80
Sblendorio, E., Dentamaro, V., Cascio, A. L., Germini, F., Piredda, M., & Cicolini, G. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105501 · Cited by 48
Li, M., Shi, T., Ziems, C., Kan, M., Chen, N. F., Liu, Z., & Yang, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.92 · Cited by 92
“LLMs struggle with named entity recognition (Ding et al., 2022; Qin et al., 2023), relational reasoning (Bang et al., 2023), affective tasks (Koco´n et al., 2023; Amin et al., 2023) and semantic similarity tasks (Kocmi and Federmann, 2023; Wang et al., 2023).”
BackgroundThelwall, M. A. (2024) · Journal of Data and Information Science · doi:10.2478/jdis-2025-0011 · Cited by 54
An Aspect-Based Review Analysis Using ChatGPT for the Exploration of Hotel Service Failures
Jeong, N., & Lee, J. (2024) · Sustainability · doi:10.3390/su16041640 · Cited by 45
Correctness Comparison of ChatGPT ‐4, Gemini, Claude‐3, and Copilot for Spatial Tasks
Hochmair, H. H., Juhász, L., & Kemp, T. (2024) · Transactions in GIS · doi:10.1111/tgis.13233 · Cited by 66
Gemini Versus ChatGPT: Applications, Performance, Architecture, Capabilities, and ImplementationPreprint
Rane, N., Choudhary, S., & Rane, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4723687 · Cited by 42
Wang, Q., & Gayed, J. M. (2024) · Computer Assisted Language Learning · doi:10.1080/09588221.2024.2371395 · Cited by 53
“Firstly, previous variants of GPT models and the downstream application ChatGPT have been extensively adopted and integrated across various language processing tasks, and many researchers have conducted studies on them (Kocon’ et al., 2023; Wu et al., 2023).”
Artificial muses: Generative Artificial Intelligence Chatbots Have Risen to Human-Level CreativityPreprint
Haase, J., & Hanel, P. H. P. (2023) · arXiv (Cornell University) · doi:10.18452/28246 · Cited by 42
A survey on chatbots and large language models: Testing and evaluation techniques
Singh, S. U., & Namin, A. S. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100128 · Cited by 83
KoSEL: Knowledge subgraph enhanced large language model for medical question answering
Zeng, Z., Cheng, Q., Hu, X., Zhuang, Y., Liu, X., He, K., & Liu, Z. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112837 · Cited by 46
Küçükkaya, A., İlaslan, E., & Göktaş, P. (2024) · Nursing Outlook · doi:10.1016/j.outlook.2024.102287 · Cited by 45
Uncovering the Reasons behind Willingness to Pay for ChatGPT-4 Premium
Jo, H. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2307692 · Cited by 41
Refashioning Emotion Recognition Modeling: The Advent of Generalized Large Models
Zhang, Z., Peng, L., Pang, T., Han, J., Zhao, H., & Schuller, B. W. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3396345 · Cited by 52
“These models are normally pretrained over large-scale corpora, and have showcased their strong capabilities in various domains and tasks, such as text generation and natural language understanding [16].”
Cheng, Y., Li, X., Xue, S., et al. (2024) · Analytical Chemistry · doi:10.1021/acs.analchem.4c03915 · Cited by 37
Ramalingam, S. P., & Kumar, V. (2024) · Building and Environment · doi:10.1016/j.buildenv.2024.112174 · Cited by 35
A novel large-language-model-driven framework for named entity recognition
Wang, Z., Chen, H., Xu, G., & Ren, M. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104054 · Cited by 40
Willms, A., & Liu, S. (2023) · JMIR Medical Education · doi:10.2196/51426 · Cited by 35
“It can understand and generate responses in various languages, as well as debug code, write stories in different genres and lengths, summarize information from complex texts, offer explanations on various topics, and even reject answering inappropriate prompts [31].”
Ta-Adapter: Enhancing few-shot CLIP with task-aware encoders
Zhang, W., Zhang, Y., Deng, Y., Zhang, W., Lin, J., Huang, B., Zhang, J., & Yu, W. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110559 · Cited by 36
Fu, C., Silalahi, A. D. K., Shih, I.-T., Phuong, D. T. T., Eunike, I. J., & Jargalsaikhan, S. (2024) · Human Behavior and Emerging Technologies · doi:10.1155/2024/8114315 · Cited by 48
Review of External Field Effects on Electrocatalysis: Machine Learning Guided Design
Wang, L., Zhou, X., Luo, Z., Liu, S., Yue, S., Chen, Y., & Liu, Y. (2024) · Advanced Functional Materials · doi:10.1002/adfm.202408870 · Cited by 38
Looi, C., & Jia, F. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13369-z · Cited by 50
“Achieving these capabilities would require significant advancements in AI, particularly in the areas of learner modeling, adaptive learning systems, and affective computing (Kocoń et al., 2023).”
Companionship in code: AI’s role in the future of human connection
Andersson, M. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-05536-x · Cited by 47
Doo, F. X., Cook, T. S., Siegel, E. L., Joshi, A., Parekh, V. S., Elahi, A., & Yi, P. H. (2023) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2023.07.007 · Cited by 32
Learning Concise and Descriptive Attributes for Visual Recognition
Yan, A., Wang, Y., Zhong, Y., Dong, C., He, Z., Lu, Y., Wang, W. Y., Shang, J., & McAuley, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00287 · Cited by 43
“ates new opportunities for building interpretable visual recognition models, as demonstrated by the powerful capabilities of models such as GPT-3 and ChatGPT in encoding world knowledge [5, 32, 21].”
Uncovering the Causes of Emotions in Software Developer Communication Using Zero-shot LLMs
Imran, M. M., Chatterjee, P., & Damevski, K. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639223 · Cited by 39
Xu, Z., Fang, Q., Huang, Y., & Xie, M. (2024) · PLoS ONE · doi:10.1371/journal.pone.0302502 · Cited by 40
“Users might engage in conversations regarding the functionalities and limitations of the API, deliberating on the prospect of integrating ChatGPT’s language generation capabilities into their applications or systems using the API [89].”
Metadata
- Authors
- Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., Kocoń, A., Koptyra, B., Mieleszczenko-Kowszewicz, W., Miłkowski, P., Oleksy, M., Piasecki, M., Radliński, Ł., Wojtasik, K., Woźniak, S., & Kazienko, P.
- Affiliations
- Wrocław University of Science and Technology (PL)
- Corresponding author
- Kocoń, Jan
- ORCID
- Venue
- Information Fusion
- Publisher
- Elsevier BV
- Published
- 2023-11
- Volume
- 99
- Pages
- 101861
- ISSN
- 1566-2535 (ISSN-L)
- OpenAlex
- W4379259169
- Semantic Scholar
- 5848737f78397f72ceae2ba6f3419a6a8502b8ba
- Cited by
- 758
- References
- 105
- Citations by year
- 2026: 95 · 2025: 234 · 2024: 313 · 2023: 115
- FWCI
- 71.55
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Information Fusion (publishedVersion, cc-by)
- SSRN Electronic Journal (acceptedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Ministerstwo Edukacji i Nauki
- Politechnika Wrocławska
- HORIZON EUROPE Framework Programme
- European Regional Development Fund (01-00-0288/22, POIR.01.01, POIR.04.02.00-00C002/19)
- European Commission Marie Sklodowska-Curie Actions (101086321)
- Department of Artificial Intelligence, Korea University
- Narodowe Centrum Nauki (2021/41/B/ST6/04471)
- European Research Executive Agency
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- ChatGPT, GPT-4, large language models, automated assessment, sentiment analysis, emotion recognition, stance detection, word sense disambiguation, linguistic acceptability, question answering, few-shot prompting, algorithmic bias, human-machine interaction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
