CoChat
Sign Up
Cited by 758Topic ModelingNatural Language Processing TechniquesSentiment Analysis and Opinion Mining

ChatGPT: Jack of all trades, master of none

Open access
Kocoń, Jan, Cichecki, Igor, Kaszyca, Oliwier, Kochanek, Mateusz, Szydło, Dominika, Baran, Joanna, Bielaniewicz, Julita, Gruza, Marcin +12 more·2023·Information Fusion
DOI 10.1016/j.inffus.2023.101861
Open this paper in CoChat — free account

Abstract

OpenAI has released the Chat Generative Pre-trained Transformer (ChatGPT) and revolutionized the approach in artificial intelligence to human-model interaction. The first contact with the chatbot reveals its ability to provide detailed and precise answers in various areas. Several publications on ChatGPT evaluation test its effectiveness on well-known natural language processing (NLP) tasks. However, the existing studies are mostly non-automated and tested on a very limited scale. In this work, we examined ChatGPT’s capabilities on 25 diverse analytical NLP tasks, most of them subjective even to humans, such as sentiment analysis, emotion recognition, offensiveness, and stance detection. In contrast, the other tasks require more objective reasoning like word sense disambiguation, linguistic acceptability, and question answering. We also evaluated GPT-4 model on five selected subsets of NLP tasks. We automated ChatGPT and GPT-4 prompting process and analyzed more than 49k responses. Our comparison of its results with available State-of-the-Art (SOTA) solutions showed that the average loss in quality of the ChatGPT model was about 25% for zero-shot and few-shot evaluation. For GPT-4 model, a loss for semantic tasks is significantly lower than for ChatGPT. We showed that the more difficult the task (lower SOTA performance), the higher the ChatGPT loss. It especially refers to pragmatic NLP problems like emotion recognition. We also tested the ability to personalize ChatGPT responses for selected subjective tasks via Random Contextual Few-Shot Personalization, and we obtained significantly better user-based predictions. Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI. Our results provide the basis for a fundamental discussion of whether the high quality of recent predictive NLP models can indicate a tool’s usefulness to society and how the learning and validation procedures for such systems should be established.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper evaluates ChatGPT on 25 analytical NLP tasks and GPT-4 on five subsets, automating prompts and analyzing more than 49k responses. It finds ChatGPT never reaches SOTA performance, with an average loss of about 25%, and that losses are larger for harder, pragmatic, and especially emotion-related tasks.

Key findings

  • ChatGPT underperformed SOTA on every task: average SOTA quality was 73.7% versus ChatGPT’s 56.5%, with an average loss of 25.5%; ChatGPT was also less stable (std. dev. 23.3% vs. 16.7%). Excluding eight emotion-related tasks, average loss fell to 12.8% (ChatGPT 69.7% vs. SOTA 80.0%).
  • Loss correlated with task difficulty (Pearson 0.46); pragmatic tasks were harder on average (32.3% difficulty) than semantic tasks (15.6%), and emotion tasks had an average loss of 52.59% (ChatGPT 28.44% vs. SOTA 60.28%).
  • Random Contextual Few-Shot Personalization improved user-based predictions: GoEmoPer3 was 4.08 percentage points better than GoEmoPer0, while AggressionPer and UnhealthyPer showed higher ChatGPT loss than their non-personalized counterparts by 3.25 and 12.55 percentage points, respectively.
  • GPT-4 had small loss on semantic tasks (0.9%–6.4% for TextEntail and WNLI) but still high loss on emotional tasks (46.3%–56.3%); ChatGPT still outperformed GPT-4 on GoEmo and PolEmo. Qualitative analysis found bias, moralizing refusals, non-determinism, and cases where experts accepted ChatGPT answers over human labels (e.g., Unhealthy: 81% ChatGPT vs. 43% human).

Methods

  • Tested 25 NLP datasets/tasks (23 English, 2 Polish), split into semantic and pragmatic categories, including sentiment, emotion, offensiveness, stance, WSD, NLI, and QA; some tasks used personalized user context.
  • Automated prompting via an unofficial PyGPT API using up to 20 accounts, with manual post-processing; the paper reports over 38,000 prompts in the setup and more than 49k responses in the abstract.
  • Compared ChatGPT to SOTA models by rerunning available source code or using reported metrics, using F1 Macro or accuracy and computing Loss = 100% × (SOTA − ChatGPT) / SOTA; GPT-4 was tested on five selected subsets.

Limitations

  • Stated: prompts may not be strict or precise enough; post-processing was sometimes required; ChatGPT produced unexpected out-of-range responses; internal policy and biases affected answers; context was generally respected but offered limited control; efficiency and costs were constrained by lack of an official API, with about 2–3k prompts per day achieved.
  • Stated: the difficulty analysis is limited because only 25 tasks were considered; GPT-4 testing was restricted by access limits (25 prompts per 3 hours).
  • Evident from the text: dataset availability for ChatGPT training was assessed manually and is uncertain; qualitative expert evaluation covered 100 randomly selected cases per dataset and involved subjective judgments; ChatGPT showed non-determinism, with results varying across launches.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1016/j.inffus.2023.101861

Cites

76 of 76 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 15 cited sources checked · 0 supported · 0 flagged

  • Findingown data
    § ABSTRACT · p. 1

    For the GPT-4 model, the loss for semantic tasks is significantly lower than for ChatGPT.

    “For GPT-4 model, a loss for semantic tasks is significantly lower than for ChatGPT.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
  • Findingown data
    § ABSTRACT · p. 1

    The more difficult the task (lower SOTA performance), the higher the ChatGPT loss.

    “We showed that the more difficult the task (lower SOTA performance), the higher the ChatGPT loss.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
  • Findingown data
    § 6.1. Jack of all trades, master of none · p. 10

    ChatGPT was less stable than SOTA: the standard deviation of its performance was 23.3% compared to 16.7% for the SOTA solutions.

    “Simultaneously, ChatGPT was less stable: the standard deviation of its performance was 23.3% compared to only 16.7% for the SOTA solutions.”
    Quote in the paperSubstantial claim
  • Findingown data
    § ABSTRACT · p. 2

    For all tested datasets, a significant drop in quality relative to state-of-the-art models was observed.

    “because for all of these datasets, we observe a significant drop in quality relative to state-of-the-art models.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
  • Findingreasoning
    § ABSTRACT · p. 1

    The higher ChatGPT loss especially refers to pragmatic NLP problems like emotion recognition.

    “It especially refers to pragmatic NLP problems like emotion recognition.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.
  • May combine two claims. 46
  • Classified as probably not a substantial claim. 70

  • May combine two claims.

    Findingown data
    § 6.1. Jack of all trades, master of none · p. 10

    ChatGPT's performance was lower than the SOTA methods (loss > 0) in all 25 NLP tasks considered, meaning it never reached the level of the best existing models.

    “The crucial finding from our studies is that the ChatGPT performance is always lower than the SOTAmethods (loss>0) in all the tasks considered.”
    • Also stated in: § 6.1. Jack of all trades, master of none · p. 10 · plain
      “It means that ChatGPT never reached the level of the best existing models.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.1. Jack of all trades, master of none · p. 10

    The loss for most tasks did not exceed 25%; it was greater only for GoEmotions, PolEmo, and TweetEmoji.

    “The loss for most tasks did not exceed 25%. It was greater only for three problems: GoEmotions, PolEmo, and TweetEmoji.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.1. Jack of all trades, master of none · p. 10

    The average quality of SOTA methods was 73.7%, whereas ChatGPT's average quality was 56.5%.

    “The average quality of SOTA methods was at 73.7%, whereas ChatGPT was at only 56.5%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.7. Comparison with GPT-4 · p. 15

    GPT-4 achieved significantly higher scores in the remaining three semantic tasks, with even a 9.9 p.p. increase for the WNLI task.

    “while GPT-4 achieved significantly higher scores in the remaining three semantic tasks with even a 9.9 p.p. increase for the WNLI task.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 6.3. Random Contextual Few-Shot Personalization · p. 12

    While solving mathematical calculations (SQuAD), the highest loss among semantic tasks was 23.7%.

    “While solving mathematical calculations (SQuAD), the highest loss was among semantic tasks: 23.7%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.2. Task difficulty vs. ChatGPT performance · p. 10

    Pragmatic tasks are more difficult (average difficulty = 32.3%), while the average difficulty for semantic tasks is only 15.6%.

    “In general, pragmatic tasks are more difficult (average difficulty = 32.3%), while the average difficulty for semantic tasks is only 15.6%.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 7.1. Exploratory analysis: Case study · p. 17

    When unable to perform a task on the provided example, ChatGPT refuses to do so and provides an explanation why.

    “We also noticed that when it is unable to perform a task on the provided example, it will refuse to do so and provide an explanation why”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingnone given
    § 7.1. Exploratory analysis: Case study · p. 15

    Such situations are usually solved by supplementing the model with information.

    “Those situations are usually solved by supplementing the model with information.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.2. Task difficulty vs. ChatGPT performance · p. 10

    Emotional tasks are pragmatic and very difficult (average 39.7%).

    “It comes especially from the emotional tasks, which are pragmatic and very difficult (average 39.7%).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 7.1. Exploratory analysis: Case study · p. 17

    In mathematical reasoning, ChatGPT can provide a whole explanation of how it reached a certain outcome or provide only the answer without explanation.

    “In the case of mathematical reasoning, it can provide a whole explanation of how it reached a certain outcome and provide only the answer without explanation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § 7.1. Exploratory analysis: Case study · p. 15

    ChatGPT can perform various tasks, including recognizing generalized and personalized dimensions of Natural Language Processing, answering questions where a generous amount of domain knowledge is required, or even writing lines of code in the programming language of choice.

    “When exploiting the possibilities of ChatGPT, we can see that it can perform various tasks, including recognizing generalized and personalized dimensions of Natural Language Processing, answering questions where a generous amount of domain knowledge is required, or even writing lines of code in the programming language of choice.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.2. Task difficulty vs. ChatGPT performance · p. 10

    The loss is correlated with task difficulty; the Pearson correlation coefficient between difficulty and loss is 0.46.

    “We can also observe that the loss is correlated with the task difficulty; see Fig. 7. The Pearson correlation coefficient between difficulty and loss is equal to 0.46.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 7.1. Exploratory analysis: Case study · p. 15

    In humor recognition or offensiveness detection experiments, ChatGPT output refused to answer whether something is or is not funny and sent a moralizing message with an irritated tone.

    “When conducting experiments regarding tasks such as humor recognition or offensiveness detection, we have stumbled upon output that not only refuses to answer whether something is or is not funny but also sends a moralizing message with an irritated tone.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § 7.1. Exploratory analysis: Case study · p. 17

    ChatGPT can return a number as a category indicator or whole output in the JSON format.

    “ChatGPT can also return a number as a category indicator or whole output in the JSON format.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.1. Jack of all trades, master of none · p. 10

    When emotion-related tasks are discarded, the average loss is reduced by half, from 25.5% to 12.8%, and the difference in performance drops from 17.2 percentage points to 10.3 percentage points.

    “In such a case, the average loss is reduced by as much as half, from 25.5% to 12.8%; the difference in performance drops from 17.2pp to 10.3pp.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 6.1. Jack of all trades, master of none · p. 10

    If all eight emotion-related tasks are discarded, average SOTA performance reaches 80% (an increase of 6.3 percentage points), while ChatGPT improves by 13.2 percentage points to 69.7%.

    “If we discard all eight emotion-related tasks (ids: 12-17, 20-21), the average SOTA performance reaches 80%(increase by 6.3pp), but ChatGPT improves much more: by 13.2pp, up to 69.7%.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 7.1. Exploratory analysis: Case study · p. 16

    ChatGPT will only generate the content if the user adequately motivates it with scientific goals.

    “ChatGPT will only generate the content if the user adequately motivates it with the scientific goals.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Findingnone given
    § 7.1. Exploratory analysis: Case study · p. 17

    For the ChatGPT model, prompt tuning is technically unavailable.

    “However, in the case of the ChatGPT model, prompt tuning is technically unavailable”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 7.1. Exploratory analysis: Case study · p. 17

    In ClarinEmo B.12, the model stated that all provided texts are legal and financial statements.

    “as it has happened in the case of ClarinEmo B.12, where the model stated that all provided texts are legal and financial statements.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 6.4. Impact of the context · p. 14

    The differences in performance are not significant.

    “Anyway, the differences in performance are not significant.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § 6.4. Impact of the context · p. 14

    ChatGPT demonstrated instability and a tendency towards non-determinism.

    “Onthe other hand, ChatGPT demonstrated its instability and tendency towards non-determinism.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § 8. Limitations and discussion · p. 21

    Manual correction of typos provided by ChatGPT is sometimes necessary.

    “It also refers to the necessary manual correction of typos surprisingly provided by ChatGPT”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § ChatGPT answer · p. 21

    Prompts themselves and the sequence of prompts provide limited control over the context respected by ChatGPT.

    “However, prompts themselves and the sequence of prompts provide limited control over them.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63: · p. 20

    ChatGPT evaluates the main content but ignores the broader context.

    “So it evaluates the main content but ignores the broader context (see Chat 64).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingreasoning
    § ChatGPT answer · p. 21

    Some ChatGPT responses are locked due to the OpenAI policy.

    “Therefore, some responses are locked due to the OpenAI policy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § 8. Limitations and discussion · p. 21

    The unlisted emotion satisfaction has been delivered.

    “e.g., the unlisted emotion satisfaction has been delivered:”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63: · p. 20

    In this example, the sentence was interpreted as not aligned with society's standards.

    “In this example, the sentence was interpreted as not aligned with society's standards.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Who is right? (expert rate) · p. 40

    In the aggression task case 520, the expert rated ChatGPT's answer as correct.

    “## Who is right? (expert rate)”
    • Also stated in: § Expected · p. 41 · plain
      “Evaluation result human - not correct, ChatGP - correct”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § Who is right? (expert rate) · p. 40

    In the GoEmo task case 7 with text 'You're welcome.', the expert rated ChatGPT's answer as correct.

    “## Who is right? (expert rate)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Expected · p. 41

    In the TweetSent task case 2776, the expert rated the human answer as incorrect.

    “Evaluation result human - not correct, ChatGP - correct”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § Who is right? (expert rate) · p. 40

    In the aggression task case 972, the expert rated ChatGPT's answer as correct.

    “## Who is right? (expert rate)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § Evaluation result · p. 44

    In the GoEmotions task (case 278), both the human answer and ChatGPT's answer were correct.

    “human - correct, ChatGP - correct”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Expected · p. 41

    In the aggression task case 520, the expert rated the human answer as incorrect.

    “Evaluation result human - not correct, ChatGP - correct”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § Table 7 · p. 19

    The annotator evaluated the message as expressing gratitude, whereas ChatGPT regards it as neutral.

    “The annotator has evaluated the below message as expressing gratitude, whereas ChatGPT regards it as neutral (Chat 58)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63: · p. 20

    ChatGPT flattens the message, partially ignoring the metatext.

    “3. ChatGPT flattens the message, partially ignoring the metatext.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingown data
    § When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63: · p. 20

    ChatGPT explicitly distances itself from judging people.

    “At the same time, ChatGPT explicitly distances itself from judging people.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given
    § Recent methods · p. 18

    ChatGPT labeled as aggressive only 11 texts from the WikiDetox Aggression dataset labeled by people as non-aggressive, while the opposite decision was taken 207 times.

    “ChatGPT labeled as aggressive only 11 texts from the WikiDetox Aggression dataset labeled by people as non-aggressive, while the opposite decision was taken 207 times.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Expected · p. 41

    In the TweetSent task case 2776, the expert rated ChatGPT's answer as correct.

    “Evaluation result human - not correct, ChatGP - correct”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given
    § Expected answer · p. 22

    Restrictive or wrongly implemented policy influences the user experience of ChatGPT by not answering all user's prompts, exaggeratedly judging many questions not along with OpenAI rules.

    “Restrictive or wrongly implemented policy will influence the user experience of the ChatGPT by not answering all user's prompts exaggeratedly judging many questions not along with OpenAI rules:”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Findingnone given
    § Expected · p. 43

    In the TweetSent task (case 2108), the human answer was correct while ChatGPT's answer was not correct.

    “human - correct, ChatGP - not correct”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63: · p. 20

    When asked which language phenomena demonstrate enthusiasm or positive sentiment in the text, ChatGPT gives a list containing such phenomena as the use of superlatives or lack of negative language.

    “when asked which language phenomena demonstrate the enthusiasm or positive sentiment in the text, it gives a list containing such phenomena as the use of superlatives or lack of negative language”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § ChatGPT answer · p. 21

    ChatGPT generally respects context.

    “Context is generally respected by ChatGPT.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § Evaluation result · p. 44

    In the Aggression task (case 477), both the human answer and ChatGPT's answer were correct.

    “human - correct, ChatGP - correct”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § ChatGPT answer · p. 21

    ChatGPT solves the problem of controversial and ambiguous utterances by demonstrating a lower level of confidence.

    “The problem of controversial and ambiguous utterances is solved by ChatGPT's demonstration of a lower level of confidence”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § Who is right? (expert rate) · p. 40

    In the GoEmo task case 7 with text 'You're welcome.', the expert rated the human expected answer as correct.

    “## Who is right? (expert rate)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § Evaluation result · p. 44

    In the GoEmotions task (case 422), neither the human answer nor ChatGPT's answer was correct.

    “human - not correct , ChatGPT - not correct”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based).

    Findingnone given
    § Who is right? (expert rate) · p. 40

    In the unhealthy task case 5, the expert rated the human answer as correct.

    “## Who is right? (expert rate)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63: · p. 20

    ChatGPT tends to find negative connotations in the given text.

    “As mentioned in the previous section, ChatGPT tends to find negative connotations in the given text.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63: · p. 20

    When asked, ChatGPT explains in detail why a phrase has a particular interpretation.

    “When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63:”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63: · p. 20

    ChatGPT justifies the negative sentiment of the news about the ban on naming streets after Fidel Castro.

    “Similarly, ChatGPT justifies the negative sentiment of the news about the ban on naming streets after Fidel Castro, Chat 66:”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § ChatGPT answer · p. 21

    Internal policy can strongly influence the answers received by ChatGPT.

    “Internal policy can strongly influence the answers received.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given
    § 6.1. Jack of all trades, master of none · p. 10

    GoEmotions, PolEmo, and TweetEmoji are related to a very subjective problem of emotional perception and individual interpretation of the content.

    “All these tasks are related to a very subjective problem of emotional perception and individual interpretation of the content.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § 2. Related work · p. 4

    In cases where ChatGPT's performance was compared to other solutions, it worked on a comparable level to the competitor but did not outperform any major SOTA solutions.

    “In cases where the performance of ChatGPT was compared to other solutions (e.g. [26, 30, 41]), it worked on a comparable level to the competitor but not outperforming any major SOTA solutions.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [26] Kung, 2022

      Not checked in the free run.

    • cites [30] Goyal, 2022

      Not checked in the free run.

    • cites [41] Jiao, 2023

      Not checked in the free run.

  • May combine two claims.

    Premisecitation
    § 2. Related work · p. 4

    Recent large-scale evaluations indicate that ChatGPT's performance is significant but it does not outperform SOTA solutions, except for sentiment analysis in one study.

    “There were a couple of large-scale evaluations published recently [46, 47] and both of them indicate that while the performance of ChatGPT is significant it doesn't outperform the SOTA solutions with the exception of sentiment analysis task in case of [46] which suggests it is a significant area for further research.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [46] Amin, 2023

      Not checked in the free run.

    • cites [47] Bang, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § ABSTRACT · p. 1

    The ChatGPT bias is most likely due to the rules imposed on human trainers by OpenAI.

    “Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 25 selected NLP tasks are representative of typical NLP analytical tasks.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisecitation
    § ABSTRACT · p. 2

    ChatGPT's successor, GPT-4, is most likely an even larger model that can additionally receive not only text but also images as input.

    “ChatGPT's successor, the GPT-4 [19], is most likely an even larger model that can additionally receive not only text but also images as input.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19] OpenAI, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § ABSTRACT · p. 2

    Language models are prone to generating responses containing human-like biases as well as presenting moral and ethical stances.

    “Language models are prone to generating responses containing human-like biases as well as presenting moral and ethical stances [23].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [23] Schramowski, 2022, Large pre-trained language models contain human-like biases of what is right and wrong to do, Nat. Mach. Intell. source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given
    § Evolution from Transformer architecture to ChatGPT · p. 3

    Even though the creators of ChatGPT secure the model against answers that are offensive, there are still multiple methods that may reveal its hidden biases.

    “Even though the creators of ChatGPT secure the model against answers that are offensive, there are still multiple methods that may reveal its hidden biases.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § 6.1. Jack of all trades, master of none · p. 10

    ChatGPT performs pretty well on all tasks except emotional ones.

    “Nevertheless, we can state that ChatGPT performs pretty well on all tasks except emotional ones.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data
    § ChatGPT evaluation flow diagram · p. 9

    Overall, the number of cases that required post-processing was relatively small; for most tasks (16), the contribution of such texts was less than 5%, and only for Aggression, SQuAD, Unhealthy, and ReAding did it exceed 15%.

    “Overall, the number of cases that required post-processing was relatively small (column #Post-processed in Tab. 1). For most tasks (16), the contribution of such texts was less than 5%. Only for Aggression, SQuAD, Unhealthy, and ReAding, it exceeded 15%.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Manual post-processing correctly extracted and mapped ChatGPT's raw outputs to the intended labels.

  • May combine two claims.

    Interpretationown data
    § 6.1. Jack of all trades, master of none · p. 10

    ChatGPT's achievements are always below SOTA but usually not by much.

    “Simultaneously, its achievements are always below SOTA but usually not so much.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § 6.1. Jack of all trades, master of none · p. 10

    The results prove that ChatGPT is a jack of all trades, master of none.

    “Such results prove that ChatGPT is Jack of all trades, master of none .”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning
    § 7.1. Exploratory analysis: Case study · p. 17

    Using prompts directly as humans designed them allows evaluation of models' language comprehension abilities.

    “On the other hand, using the prompts directly as humans designed them implicitly allows us to evaluate models' language comprehension abilities.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning
    § 6.7. Comparison with GPT-4 · p. 15

    Results for GPT-4, as for ChatGPT, could significantly differ if distinct prompt schemes are compared.

    “We emphasize that the results for GPT-4, as for ChatGPT, could significantly differ, if distinct prompt schemes are compared, see Sec. 6.6.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning
    § 5.4. Performance measures · p. 9

    If possible, the paper launched models equivalent to SOTA solutions; for some tasks, they exploited performance results from the original papers.

    “If possible, we launched our models equivalent to SOTA solutions since the setup (especially data split) was often different than in the original paper. For that purpose, we usually utilized source codes published by the authors. Unfortunately, it was impossible for some tasks, so we exploited the performance results provided in the original paper.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The SOTA results taken from original papers are comparable to the paper's own setup despite differences in data split and evaluation.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 7.1. Exploratory analysis: Case study · p. 17

    Borji conducted a systematic review of the typical categories of ChatGPT failures.

    “Borji [105] conducted a systematic review of the typical categories of ChatGPT failures.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [105] Korczyński, 2022, Compression methods for transformers in multidomain sentiment analysis

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 7.1. Exploratory analysis: Case study · p. 17

    The above errors are derived from both incorrect reasoning in terms of psychological reasoning and bias and discrimination.

    “The above errors are derived from both incorrect reasoning in terms of psychological reasoning and bias and discrimination.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Study design and conductown data
    § 6.2. Task difficulty vs. ChatGPT performance · p. 10

    The analysis requires further investigations because the number of tasks considered (25) remains relatively small.

    “This analysis, however, requires further investigations since the number of the tasks considered (25) still remains relatively small.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 6.2. Task difficulty vs. ChatGPT performance · p. 10

    Semantic tasks are rather easy; hence, their ChatGPT loss is relatively small.

    “It is observable that semantic tasks (blue crosses) are rather easy; hence, their ChatGPT loss is relatively small.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § Table 1 · p. 7

    The annotations of six independent annotators were aggregated to label each sentence with all potential options, using the label when at least two annotators agreed on it.

    “The annotations of six independent annotators were aggregated to label each sentence with all potential options, using the label when at least two annotators agreed on it.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § Table 1 · p. 8

    Using the glosses to explain senses to a language model implicitly tests its language comprehension abilities.

    “Using the glosses to explain senses to a language model implicitly tests its language comprehension abilities.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data
    § Table 1 · p. 7

    The ClarinEmo dataset is the authors' new dataset that has not yet been published, and they exploited it to ensure that ChatGPT was not trained on it.

    “It is our new dataset that has not yet been published. We exploited this dataset to ensure that ChatGPT was not trained on it.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ChatGPT was not trained on the ClarinEmo dataset.

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 7.1. Exploratory analysis: Case study · p. 17

    The only way to verify prompt relevance is to evaluate its performance directly in the downstream task.

    “and the only way to verify prompt relevance is to evaluate its performance directly in the downstream task.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning
    § 5.4. Performance measures · p. 9

    The paper used F1 Macro as the primary measure if available, because it is commonly acceptable for imbalanced data.

    “If available, we tried to validate ChatGPT using one measure F1 Macro, which is commonly acceptable for imbalanced data, Tab. 2.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning
    § 5.4. Performance measures · p. 10

    For CoLa, WNLI, WordContext, and MathQA, the paper had to rely on accuracy because it was the only measure presented in the reference paper; they could not replicate the studies and calculate their own measures. WNLI and WordContext have balanced classes, so this is not an issue.

    “In the case of CoLa, WNLI, WordContext, and MathQA, we had to rely on the accuracy, as it was the only one presented in the reference paper; we could not replicate their studies and calculate our measures. WNLI and WordContext have their two classes balanced, so it is not an issue.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning
    § 7.1. Exploratory analysis: Case study · p. 15

    This implies possible consequences regarding clashes with fake news.

    “which further implies possible consequences regarding clashes with fake news.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 6.4. Impact of the context · p. 14

    Providing some answerable questions may help ChatGPT detect unanswerable ones with the same context.

    “It may suggest that providing some answerable questions helps it detect unanswerable ones with the same context.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § 6.4. Impact of the context · p. 12

    Demonstration-based personalization included in the prompts can be treated as similar to few-shot learning, even an extended context to a given prompt.

    “Demonstration-based personalization included in our prompts can be treated as similar to few-shot learning, even an extended context to a given prompt.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 7.1. Exploratory analysis: Case study · p. 16

    Chatbot ethics can be a subject of debate in fairness, nondiscrimination, and justice.

    “Hagendorff [104] drew attention to the fact that chatbot ethics can be a subject of debate in fairness, nondiscrimination, and justice.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [104] Borji, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 8. Limitations and discussion · p. 21

    Raw text provided by ChatGPT may require extraction of crucial information, e.g., final labels.

    “As a result, the raw text provided by ChatGPT may require extraction of crucial information, e.g., final labels.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisereasoning
    § When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63: · p. 20

    Information about the distinction for a particular footballer is neutral.

    “Information about the distinction for a particular footballer is neutral.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Study design and conductreasoning
    § 8. Limitations and discussion · p. 21

    Prompts may not be strict and precise enough.

    “1. Prompts may not be strict and precise enough.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data
    § When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63: · p. 20

    ChatGPT seems to have no regard for individuals, instead judging situations.

    “2. ChatGPT seems to have no regard for individuals, instead judging situations.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63: · p. 20

    This often leads to mistakes.

    “However, this often leads to mistakes”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data
    § When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63: · p. 20

    ChatGPT presents the sense of common human morality.

    “6. ChatGPT presents the sense of common human morality.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given
    § 8. Limitations and discussion · p. 21

    ChatGPT requires prompts in natural language, which is by nature not structured and can be different for different users and tasks.

    “ChatGPTrequires prompts in natural language, which is - by nature - not structured and can be different for different users and tasks.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given
    § A. Additional results · p. 27

    A small difference between the entropy of the test/dev set and the entropy of the subset used for prompting indicates a similar class distribution in both sets, demonstrating good stratification of the sampling.

    “A small difference in these two values proves a similar distribution of classes in both sets, thus, a good stratification of sampling.”
    • Also stated in: § Table 8 · p. 28 · boosted
      “A marginal difference in entropy of output real values for test or dev set (column #Test in Tab. 1) and within the set used by us (column #Used in Tab. 1) demonstrates a good stratification of the used set selection.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § 8. Limitations and discussion · p. 21

    Post-processing depends on prompting quality and correctness.

    “It also depends on the prompting quality and correctness.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63: · p. 20

    Its sentiment can be both positive and negative, depending on the sympathies of the recipient regarding specific footballers.

    “Its sentiment, however, can be both positive and negative. It depends on the sympathies of the recipient sympathies regarding specific footballers.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § When asked, ChatGPT explains in detail why a phrase has a particular interpretation, Chat 63: · p. 20

    ChatGPT evaluates situations rather than participants, but words refer to people, which leads to a specific, predetermined assessment.

    “ChatGPT evaluates rather situations than participants, but words refer to people, which lead to a specific, predetermined assessment”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § Recent methods · p. 18

    Analysis was conducted by experts who are specialists trained in the recognition of emotions in the text; one is a psychologist and another is a linguist, both experienced annotators.

    “Analysis was conducted by experts who are specialists trained in the recognition of emotions in the text. One of them is a psychologist and another is a linguist, both are experienced annotators.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § 10. Conclusions and future work · p. 22

    ChatGPT provides a unique self-explanation capability that facilitates human understanding and adaptation to the expected outcome.

    “It also provides a unique selfexplanation capability that facilitates human understanding and adaptation to the expected outcome.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § 7.3. Explanatory analysis: XAI · p. 20

    ChatGPT is self-explanatory artificial intelligence, which is a part of eXplainable Artificial Intelligence (XAI).

    “Thus, we are dealing with self-explanatory artificial intelligence, which is a part of eXplainable Artificial Intelligence (XAI)”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given
    § ChatGPT answer · p. 21

    The general ChatGPT model incorporates various biases and political correctness.

    “It includes various biases and political correctness incorporated into the general ChatGPT model.”
    Quote in the paperNot a substantial claim
94 of 94

  • The paper relies on this without support.

    The 25 selected NLP tasks are representative of typical NLP analytical tasks.

    “We tested ChatGPT on 25 tasks focusing on solving common NLP problems and requiring analytical reasoning, Tab. 1.”

  • The paper relies on this without support.

    ChatGPT's self-reports about whether it used a dataset for training are reliable.

    “Additionally, we asked ChatGPT whether or not the dataset was used for training.”

  • The paper relies on this without support.

    ChatGPT was not trained on the ClarinEmo dataset.

    “It is our new dataset that has not yet been published. We exploited this dataset to ensure that ChatGPT was not trained on it.”

  • The paper relies on this without support.

    The unofficial PyGPT API and the use of up to 20 accounts produced outputs equivalent to the official ChatGPT service.

    “Without an official API, we modified and used an unofficial API called PyGPT 5 , written in Python. During the research, we exploited up to 20 accounts to gather data regarding 25 datasets.”

  • The paper relies on this without support.

    The SOTA results taken from original papers are comparable to the paper's own setup despite differences in data split and evaluation.

    “If possible, we launched our models equivalent to SOTA solutions since the setup (especially data split) was often different than in the original paper. For that purpose, we usually utilized source codes published by the authors. Unfortunately, it was impossible for some tasks, so we exploited the performance results provided in the original paper.”

  • The paper relies on this without support.

    Manual post-processing correctly extracted and mapped ChatGPT's raw outputs to the intended labels.

    “This imposes a necessity to check what happened and extract answers from ChatGPT output manually. The next step is to cast the resulting outputs to the correct labels in the dataset.”

  • The paper relies on this without support.

    The performance of modern language models such as T5, GPT-3, and ChatGPT heavily relies on the quality of task-specific prompts.

    “The performance of modern language models, such as T5, GPT-3, and ChatGPT, heavily relies on the quality of task-specific prompts.”

  • The paper relies on this without support.

    The prompt-based learning paradigm requires careful prompt engineering and prompt tuning.

    “The prompt-based learning paradigm requires careful prompt engineering and prompt tuning.”

  • The paper relies on this without support.

    Most tasks require a prompt that enables the model to choose a certain value from the provided options.

    “Most tasks require a prompt that enables the model to choose a certain value from the provided options.”

  • The paper relies on this without support.

    The annotations in the selected 4 datasets were of a more or less subjective nature, and detailed guidelines were not necessary to achieve high interannotator agreement.

    “The annotations in the selected 4 datasets were of a more or less subjective nature, and for this reason, it was not necessary to create detailed guidelines in order to achieve high interannotator agreement”
10 of 10

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In a large study of ChatGPT with 25 tasks and over 48,000 prompts, the tool performed reasonably well on quantitative analysis, with somewhat poorer performance on tasks requiring more reasoning.

    “In a large study of ChatGPT that included 25 tasks and over 48,000 prompts, the tool was found to perform reasonably well when conducting quantitative analysis, with a somewhat poorer performance with tasks that require more reasoning ( Kocoń et al., 2023).”

    From Generative AI for corpus approaches to discourse studies: A critical evaluation of ChatGPT (2024) · cites this paper as Kocoń et al., 2023

    The Tasks section (p. 5) confirms that the study tested ChatGPT on 25 tasks, and Task difficulty vs. ChatGPT performance (p. 10) reports greater performance losses on more difficult tasks. But Experimental setup (p. 9) says the study used over 38,000 prompts, not over 48,000. The paper evaluates NLP tasks, including a mathematical-reasoning task; it does not establish the claimed contrast between reasonably good quantitative analysis and poorer reasoning performance. (LLM, conf=0.89)

    Experimental — accuracy not yet measured

  • This underperformance by the chat model is consistent with findings by Kocon et al.

    “This underperformance by the chat model is consistent with the findings by Kocon et al. [39].”

    From Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization) (2024) · cites this paper as [39]

  • LLMs' performance in logic and deductive tasks is often poor.

    “Similarly, LLMs’ performance in logic and deductive tasks is often poor (64, 65), so using them for analysis may lead to false conclusions.”

    From How should the advancement of large language models affect the practice of science? (2025) · cites this paper as [64]

  • LLMs may struggle with complex, subjective tasks such as emotion recognition.

    “Kocoń et al. (76) found that LLMs may struggle with complex, subjective tasks such as emotion recognition, and Huang et al. (18) found that LLMs can reinforce lay perspectives when detecting implicit hate speech and may thus mislead nonexperts with their explanations in cases where they make incorrect decisions.”

    From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [76]

  • Recent critiques emphasize limitations of LLMs in text analysis, such as inconsistencies in text annotations, difficulties in explaining complex constructs such as implicit hate speech, and a potential lack of depth in specialized or sensitive areas.

    “However, this perspective of LLMs, as a convenient and comprehensive tool in psychological text analysis, is challenged by recent critiques emphasizing their limitations, such as inconsistencies in text annotations, difficulties in explaining complex constructs such as implicit hate speech, and a potential lack of depth in specialized or sensitive areas (18, 76, 77).”

    From Perils and opportunities in using large language models in psychological research (2024) · cites this paper as [76]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • AggressionPer difficulty is one point too high

    Minor

    Difficulty is defined as 100% minus SOTA performance. A SOTA score of 81.03% gives 18.97%, not 19.97%.

    Reported 19.97 · recomputed 18.97

  • ReAding SOTA score is attributed to the wrong metric and source

    Minor

    Table 2 calls the 84.71% ReAding SOTA result an in-house F1 Macro score. Table 9 lists no in-house SOTA result or paper F1 score; it lists 84.71% only as paper accuracy.

  • TextEntail comparison labels accuracy values as F1 Macro

    Minor

    Table 2 presents TextEntail's 88.09% ChatGPT and 92.10% SOTA scores as F1 Macro. Table 9 identifies those values as accuracy, gives ChatGPT F1 as 87.88%, and provides no SOTA-paper F1 score; the reported loss therefore compares accuracy scores while labeling them F1 Macro.

3 of 3

Cites

1 retracted

Flagged first, then by citations

29 of the 105 references couldn’t be matched to a known paper.

Research works printed but not linked: 6

  1. Johnson, 2016, Supervised and semi-supervised text categorization using LSTM for region embeddings, vol. 48, 526
  2. Gillioz, 2020, Overview of the transformer-based models for NLP tasks, vol. 21, 179
  3. Radford, 2018, Improving language understanding by generative pre-training, OpenAI
  4. Radford, 2019, Language models are unsupervised multitask learners, OpenAI
  5. Ngo, 2022, StudEmo: A non-aggregated review dataset for personalized emotion recognition, 46
  6. Fellbaum, 1998, Wordnet: An electronic lexical database, Comput. Linguist., 292

Cited by

Sorted by citations

Related

Authors
Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., Kocoń, A., Koptyra, B., Mieleszczenko-Kowszewicz, W., Miłkowski, P., Oleksy, M., Piasecki, M., Radliński, Ł., Wojtasik, K., Woźniak, S., & Kazienko, P.
Affiliations
Wrocław University of Science and Technology (PL)
Corresponding author
Kocoń, Jan
ORCID
Venue
Information Fusion
Publisher
Elsevier BV
Published
2023-11
Volume
99
Pages
101861
ISSN
1566-2535 (ISSN-L)
DOI
10.1016/j.inffus.2023.101861
OpenAlex
W4379259169
Semantic Scholar
5848737f78397f72ceae2ba6f3419a6a8502b8ba
Cited by
758
References
105
Citations by year
2026: 95 · 2025: 234 · 2024: 313 · 2023: 115 · 2022: 1
FWCI
71.55
Type
journal article
Language
en
Open access
Licence
Locations
  • Information Fusion (publishedVersion, cc-by)
  • SSRN Electronic Journal (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Ministerstwo Edukacji i Nauki
  • Politechnika Wrocławska
  • HORIZON EUROPE Framework Programme
  • European Regional Development Fund (01-00-0288/22, POIR.01.01, POIR.04.02.00-00C002/19)
  • European Commission Marie Sklodowska-Curie Actions (101086321)
  • Department of Artificial Intelligence, Korea University
  • Narodowe Centrum Nauki (2021/41/B/ST6/04471)
  • European Research Executive Agency
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
ChatGPT, GPT-4, large language models, automated assessment, sentiment analysis, emotion recognition, stance detection, word sense disambiguation, linguistic acceptability, question answering, few-shot prompting, algorithmic bias, human-machine interaction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-07.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-07.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review