CoChat
Sign Up
PreprintCited by 10Emotion and Mood RecognitionEthics and Social Impacts of AIDigital Mental Health Interventions

Skill but not Effort Drive GPT Overperformance over Humans in Cognitive Reframing of Negative Scenarios

Open access
Li, Joanna Z., Herderich, Alina, Goldenberg, Amit·2024
DOI 10.31234/osf.io/fzvd8
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent advancements in large language models (LLMs), such as GPT, have led to their implementation in tasks involving emotional support. However, LLM performance has not been compared to humans in both quality and the type of content produced. We examined this question by focusing on the skill of reframing negative situations to reduce negative emotions, also known as cognitive reappraisal. We trained both humans (N= 601) and GPT-4 to reframe negative vignettes (Nreappraisals = 4195) and compared their performance using human raters (N = 1744). GPT-4 outperformed humans on three of the four examined metrics. We investigated whether the gap was driven by effort or skill by incentivizing participants to produce better reappraisals, which led to increased time spent on reappraisals but did not decrease the gap between humans and GPT-4. Content analysis suggested that high-quality reappraisals produced by GPT-4 were associated with being more semantically similar to the emotional scenarios, which pointed to the fact that GPT-4's success is predicated on tuning into the specific scenario. Results were in the opposite direction for humans, whose reappraisals were rated higher when being more semantically different from the emotional scenario, suggesting that human success is predicated on generalizing away from the specific situations. These results help us understand the nature of emotional support by LLMs and how it compares to humans.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.31234/osf.io/fzvd8

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent generation large language models excel, and in many cases outpace humans, at writing persuasively and empathetically, at inferring user traits from text, and at mimicking human-like conversation believably and effectively.

    “A growing body of research suggests that the recent generation of large language model (LLMs) excel, and in many cases outpace humans, at writing persuasively and empathetically, at inferring user traits from text, and at mimicking human-like conversation believably and effectively—without possessing any true empathy or social understanding.”

    From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [56]

    The Discussion (page unavailable) reports that GPT-4 outperformed humans on empathy ratings for cognitive reappraisals of negative scenarios. That supports the empathetic-writing comparison, but the paper does not establish superiority at persuasive writing, inferring user traits, or believable human-like conversation. It also does not test whether LLMs possess true empathy or social understanding. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • LLMs outperform humans in reframing written scenarios to reduce negative emotion, a skill known in humans as cognitive reappraisal.

    “Similarly, LLMs have been shown to outperform humans in reframing written scenarios to reduce negative emotion, a skill known in humans as cognitive reappraisal (56).”

    From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [56]

  • Generative AI models are increasingly achieving or surpassing human performance benchmarks in various domains, including medical diagnostic dialogue, persuasive communication, theory of mind, making people feel heard, responding to relationship issues and helping people reframe negative situations to reduce negative emotions.

    “These models are increasingly achieving or surpassing human performance benchmarks in various domains, including medical diagnostic dialogue28, persuasive communication29, theory of mind30, making people feel heard31, responding to relationship issues32 and helping people reframe negative situations to reduce negative emotions33.”

    From “It happened to be the perfect thing”: experiences of generative AI chatbots for mental health (2024) · cites this paper as [33]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Li, J. Z., Herderich, A., & Goldenberg, A.
Affiliations
Harvard University (US)
ORCID
Publisher
Center for Open Science
Published
2024-04-19
DOI
10.31234/osf.io/fzvd8
OpenAlex
W4394977886
Cited by
10
References
43
Citations by year
2026: 4 · 2025: 3 · 2024: 3
Type
posted content
Language
en
Open access
Licence
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Social Sciences › Psychology › Applied Psychology › Digital Mental Health Interventions
Keywords
GPT-4, large language models, cognitive reappraisal, cognitive reframing, negative emotions, emotional support, human-AI comparison, semantic similarity, generalization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review