CoChat
Sign Up
Cited by 242Natural Language Processing TechniquesTopic ModelingSpeech and dialogue systems

Is chatGPT a good nlg evaluator? A preliminary study

Open access
Wang, Jiaan, Liang, Yunlong, Meng, Fandong, Sun, Zengkui, Shi, Haoxiang, Li, Zhixu, Xu, Jinan, Qu, Jianfeng +1 more·2023·Proceedings of the 4th New Frontiers in Summarization Workshop
DOI 10.18653/v1/2023.newsum-1.1
Open this paper in CoChat — free accountOpen PDF

Abstract

Recently, the emergence of ChatGPT has attracted wide attention from the computational linguistics community.Many prior studies have shown that ChatGPT achieves remarkable performance on various NLP tasks in terms of automatic evaluation metrics.However, the ability of ChatGPT to serve as an evaluation metric is still underexplored.Considering assessing the quality of natural language generation (NLG) models is an arduous task and NLG metrics notoriously show their poor correlation with human judgments, we wonder whether Chat-GPT is a good NLG evaluation metric.

AI snapshot

Written from the paper’s full text.

TL;DR

This preliminary meta-evaluation prompts ChatGPT as an NLG evaluator with task- and aspect-specific instructions and tests its correlation with human judgments on five meta-evaluation datasets. It finds ChatGPT achieves state-of-the-art or competitive correlations in most cases, but its effectiveness is prompt-sensitive and can be undermined by reference-oriented dataset biases.

Key findings

  • On SummEval, ChatGPT achieves new state-of-the-art correlation in most aspects; ChatGPT (Stars w/ ref) reaches average sample-level Spearman 0.411 (Table 1).
  • On OpenMEVA story generation, ChatGPT achieves the best performance; ChatGPT (DA w/o ref) reaches 0.507 sample-level Spearman correlation, while PPL reaches only 0.324.
  • On RealSumm, ChatGPT underperforms similarity metrics; ChatGPT (Stars w/ ref) reaches 0.195 sample-level Spearman correlation versus ROUGE-1 at 0.498, which the paper attributes to reference-oriented pyramid annotation bias.
  • On NewsRoom, ChatGPT outperforms ROUGE and BERTScore by a large margin; ChatGPT (DA w/o ref) reaches average sample-level Spearman 0.504 versus ROUGE-1 at 0.119 and BERTScore at 0.152. On BAGEL, ChatGPT obtains competitive correlations.

Methods

  • Meta-evaluation on five NLG meta-evaluation datasets: SummEval, NewsRoom, and RealSumm for summarization; OpenMEVA-ROC for story generation; BAGEL for data-to-text generation.
  • ChatGPT is prompted as a human evaluator with task-specific and aspect-specific instructions, using reference-free and reference-based settings, and direct assessment (0–100) or one-to-five stars prompts.
  • Compared against ROUGE-1/2/L, BERTScore, MoverScore, PRISM, BARTScore variants, and PPL; correlations are measured with Spearman, Pearson, and Kendall’s Tau at sample and dataset levels.

Limitations

  • Stated: ChatGPT’s performance is prompt-sensitive; experiments miss mainstream NLG tasks such as dialogue and report generation; no official API was available, so experiments used the ChatGPT website with default temperature and are difficult to reproduce; only English datasets are used; correlation depends on the quality and challenge of the meta-evaluation datasets, which are traditional pre-LLM datasets.
  • Evident from the text: this is a preliminary report with limited task and dataset coverage, and the lack of an official API/default temperature implies non-deterministic, version-dependent results.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.18653/v1/2023.newsum-1.1

Cites

45 of 45 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 7 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    All similarity-based metrics (ROUGE-1, ROUGE-2, ROUGE-L, BERTScore and BARTScore) show weak correlations with human judgments.

    “In addition, we also find that all similarity-based metrics (i.e., ROUGE-1, ROUGE-2, ROUGE-L, BERTScore and BARTScore) show their weak correlations with the human judgments.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
  • Findingown data

    ChatGPT achieves competitive correlations compared with previous state-of-the-art baselines on BAGEL data-to-text generation.

    “Table 7 shows the experimental results, where ChatGPT achieves competitive correlations compared with the previous state-of-the-art baselines, indicating its strong potentiality serving as a metric for data-to-text generation.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
  • Findingnone given

    The performance of ChatGPT as an NLG evaluator is sensitive to the format of the prompt.

    “Besides, the performances of ChatGPT as an NLG evaluator are sensitive to the format of the prompt, for different tasks and aspects, the prompt should be carefully designed.”
    • Also stated in: § Limitations · plain
      “(1) Chat-GPT's performance as an NLG metric is related to prompts, and future work could explore more powerful prompts to achieve better performance;”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
  • Findingown data

    The effectiveness of the ChatGPT evaluator might be influenced by the creation method of the meta-evaluation datasets.

    “In addition, we find that the effectiveness of the ChatGPT evaluator might be influenced by the creation method of the meta-evaluation datasets.”
    Quote in the paperSubstantial claim
  • Findingown data

    ChatGPT achieves a new state-of-the-art correlation in most aspects of SummEval.

    “Experimental results show that ChatGPT achieves a new state-of-the-art correlation in most aspects of SummEval, demonstrating its potential of serving as an NLG metric.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 12
  • May combine two claims. 12

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The finding of prompt sensitivity on OpenMEVA is consistent with that in text summarization (Section 4.3).

    “This finding is also consistent with that in text summarization (Section 4.3).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.

  • May combine two claims.

    Findingown data

    The ChatGPT evaluator has a high correlation with humans in most cases, especially for creative NLG tasks (e.g., story generation) where multiple generations can satisfy humans.

    “We find that the ChatGPT evaluator has a high correlation with humans in most cases, especially for creative NLG tasks (e.g., story generation) where multiple generations can satisfy humans.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Lexical biases involved in the meta-evaluation datasets would influence the effectiveness of NLG metrics, and might lead to the limited performance of the ChatGPT evaluator.

    “In addition, we also find that the lexical biases involved in the meta-evaluation datasets would influence the effectiveness of NLG metrics, and might lead to the limited performance of the ChatGPT evaluator.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.

  • May combine two claims.

    Findingown data

    ChatGPT (DA w/o ref) achieves 0.507 sample-level Spearman correlation, while PPL achieves 0.324 sample-level Spearman correlation on OpenMEVA.

    “For example, ChatGPT (DA w/o ref) achieves 0.507 sample-level Spearman correlation, while PPL only achieves 0.324 sample-level Spearman correlation.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.

  • May combine two claims.

    Findingown data

    ChatGPT achieves the best performance in terms of all correlations on the OpenMEVA dataset, and significantly outperforms the second-best metric (PPL).

    “As shown in Table 6, ChatGPT achieves the best performance in terms of all correlations, and significantly outperforms the second-best metric (i.e., PPL).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.

  • May combine two claims.

    Findingown data

    ChatGPT (Stars w/ ref) achieves 0.195 sample-level Spearman correlation on RealSumm, far behind ROUGE-1's 0.498.

    “For example, ChatGPT (Stars w/ ref) achieves 0.195 sample-level Spearman correlation, which is far behind the counterpart of ROUGE-1 (i.e., 0.498).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    The involved biases of the NLG meta-evaluation datasets also influence the effectiveness of NLG metrics, and might lead to the limited effectiveness of the ChatGPT evaluator.

    “We find that the involved biases of the NLG metaevaluation datasets also influence the effectiveness of NLG metrics, and might lead to the limited effectiveness of the ChatGPT evaluator.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ChatGPT exhibits a high correlation with human judgment in most cases, especially for the story generation task.

    “Experimental results show that ChatGPT exhibits a high correlation with human judgment in most cases especially for the story generation task, indicating its potential as an NLG metric.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    The experiments were conducted on the ChatGPT website with default temperature, making the results difficult to reproduce.

    “(3) When we did the experiments, the OpenAI ChatGPT did not release the official API.Thus, we conducted the experiments on the ChatGPT website with default temperature, making the results difficult to reproduce.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Our experiments are conducted on the traditional NLG meta-evaluation datasets (that appear before the LLM era).

    “Our experiments are conducted on the traditional NLG meta-evaluation datasets (that appear before the LLM era).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The experiments are only conducted on the English NLG meta-evaluation datasets.

    “(4) The experiments are only conducted on the English NLG meta-evaluation datasets, and future work could extend this method into other languages or cross-lingual scenes.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Newsroom provides human judgments on 7 model-generated summaries, including coherence, relevance, informativeness and fluency.

    “Newsroom, as a text summarization dataset, also provides human judgments on 7 model-generated summaries, including coherence, relevance, informativeness and fluency.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Grusky et al., 2018

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The reference-oriented annotation method makes traditional n-gram-based metrics (such as ROUGE) already achieve good correlations with human judgments, which the authors name lexical biases.

    “Therefore, this reference-oriented annotation method makes the traditional n-grambased metric (such as ROUGE) already achieve well correlations with human judgments, which we named as lexical biases.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The pyramid method first requires human evaluators to extract semantic content units from golden references, and then score each system summary based on how many extracted semantic content units are mentioned in the system summary.

    “In detail, this method first requires human evaluators to extract semantic content units from golden references, and then score each system summary based on how many extracted semantic content units are mentioned in the system summary.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Nenkova and Passonneau, 2004

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The correlation between the ChatGPT evaluator and humans is related to the quality and challenge of the corresponding meta-evaluation datasets.

    “(5) The correlation between the ChatGPT evaluator and humans is also related to the quality and challenge of the corresponding meta-evaluation datasets.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.

  • Classified as probably not a substantial claim.

    Premisecitation

    OpenMEVA-ROC manually annotates five model-generated storylines under the consideration of their overall quality.

    “The OpenMEVA-ROC dataset manually annotates five model-generated storylines under the consideration of their overall quality.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Guan et al., 2021

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    If a meta-evaluation dataset is created greatly depending on the reference, the similarity between model generation and references serves as a strong signal to reflect human judgments, where simple similarity-based metrics (e.g., ROUGE) can achieve very strong performance.

    “If a meta-evaluation dataset is created greatly depending on the reference, the similarity between model generation and references serves as a strong signal to reflect human judgments, where simple similarity-based metrics (e.g., ROUGE) can achieve very strong performance.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Our experiments only estimate the lower bound of ChatGPT's performance.

    “Note that our experiments only estimate the lower bound of ChatGPT's performance, and better performances would like to be achieved once using better prompts or updated versions of ChatGPT.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    This preliminary report misses experiments on some mainstream NLG tasks, e.g., dialogue generation and report generation.

    “(2) This preliminary report misses experiments on some mainstream NLG tasks, e.g., dialogue generation and report generation;”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    SummEval collects 16 model-generated summaries on the CNN/DM dataset and annotates human judgments upon these summaries covering aspects of coherence, relevance, consistency and fluency.

    “SummEval collects 16 modelgenerated summaries on the CNN/DM dataset and annotates human judgments upon these summaries covering aspects of coherence, relevance, consistency and fluency.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Fabbri et al., 2021

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    RealSumm evaluates the pyramid recall of 25 model-generated summaries.

    “RealSumm evaluates the pyramid (Nenkova and Passonneau, 2004) recall of 25 model-generated summaries.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Bhandari et al., 2020

      Not checked in the free run.

    • cites Nenkova and Passonneau, 2004

      Not checked in the free run.

21 of 21

  • The paper relies on this without support.

    The ability of ChatGPT to serve as an evaluation metric is still underexplored.

    “However, the ability of ChatGPT to serve as an evaluation metric is still underexplored.”

  • The paper relies on this without support.

    The automated assessment of the general generation quality of NLG models still remains underexplored.

    “However, the automated assessment of the general generation quality of NLG models still remains underexplored.”

  • The paper relies on this without support.

    The human judgments in the meta-evaluation datasets are reliable enough to serve as ground truth for evaluating NLG metrics.

    “SummEval collects 16 modelgenerated summaries on the CNN/DM dataset and annotates human judgments upon these summaries covering aspects of coherence, relevance, consistency and fluency.”

  • The paper relies on this without support.

    The five meta-evaluation datasets are representative of NLG tasks and suitable for evaluating general NLG metrics.

    “We conduct experiments on five NLG metaevaluation datasets (including summarization, story generation and data-to-text tasks).”

  • The paper relies on this without support.

    For a given beginning of a story, various generated storylines and different plots could satisfy people.

    “for a given beginning of a story, various generated storylines and different plots could satisfy people.”

  • The paper relies on this without support.

    Human judgments are a valid gold standard for evaluating NLG metrics.

    “ChatGPT achieves the new state-of-theart correlations (with human judgments) on Sum-mEval and OpenMEVA meta-evaluation datasets”

  • The paper relies on this without support.

    The task-specific and aspect-specific prompts used are adequate to elicit ChatGPT's NLG evaluation capability.

    “To this end, we design task-specific as well as aspectspecific prompts to guide ChatGPT to perform as an NLG metric.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • GPT-3.5 has a positive correlation with human annotations.

    “GPT-3.5 has been proved to have positive correlation with human annotations (Chiang and Lee, 2023a;Wang et al., 2023a)”

    From Bayesian Calibration of Win Rate Estimation with LLM Evaluators (2024) · cites this paper as Wang et al., 2023a

    Section 4.4 (Story Generation; page not supplied) reports a positive 0.507 sample-level Spearman correlation for ChatGPT as an evaluator, supporting the correlation with human judgments. The paper identifies that evaluator as ChatGPT but does not establish in the retrieved passages that it was specifically GPT-3.5. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Large language models can serve as reliable evaluators for natural language generation tasks, exhibiting strong correlation with human judgments.

    “Recent studies have demonstrated that large language models (LLMs) can serve as reliable evaluators for natural language generation (NLG) tasks, exhibiting strong correlation with human judgments.”

    From Enhancing memory retrieval in generative agents through LLM-trained cross attention networks (2025) · cites this paper as Wang et al. (2023)

  • Some LLM-based evaluators show good human alignment in more NLG tasks, especially compared with traditional metrics.

    “Some other evaluators based on LLMs [37,60,74,118] also show good human alignment in more NLG tasks, especially compared with traditional metrics.”

    From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [118]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • OpenMEVA result attributes the stars score to direct assessment

    Major

    The text assigns the 0.507 sample-level Spearman correlation to ChatGPT (DA w/o ref). In Table 6, the merged rows list 0.472 for DA w/o ref and 0.507 for Stars w/o ref.

  • NewsRoom ROUGE-2 average Kendall correlation is incorrect

    Minor

    The four aspect-level Kendall correlations average to 0.0395, not the reported 0.092.

    Reported 0.092 · recomputed 0.0395

  • Correlation-measure count contradicts the enumerated list

    Minor

    The paper says it adopts two correlation measures but enumerates three: Spearman, Pearson, and Kendall's Tau.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Mavuto M Mukaka. 2012. A guide to appropriate use of correlation coefficient in medical research. Malawi medical journal, 24(3):69-71.
  2. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computa- tional Linguistics, ACL '02, page 311-318, USA. Association for Computational Linguistics.
  3. Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023a. Cross-lingual summarization via chatgpt. arXiv preprint.

Cited by

Sorted by citations

Related

Authors
Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J.
Affiliations
  • Soochow University (CN)
  • Beijing Jiaotong University (CN)
  • Tencent (China) (CN)
  • Waseda University (JP)
  • Fudan University (CN)
ORCID
Venue
Proceedings of the 4th New Frontiers in Summarization Workshop
Publisher
Association for Computational Linguistics
Published
2023
Pages
1-11
DOI
10.18653/v1/2023.newsum-1.1
OpenAlex
W4389519239
Cited by
235
References
42
Citations by year
2026: 43 · 2025: 65 · 2024: 75 · 2023: 52
FWCI
41.64
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
Workshop on New Frontiers in Summarization (NewSum) (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
ChatGPT, natural language generation, automatic evaluation metrics, large language models, human judgment correlation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review