The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; none of the 7 sources its claims cite is checked for support yet.
Is ChatGPT a Good NLG Evaluator? A Preliminary Study
Open accessAbstract
Recently, the emergence of ChatGPT has attracted wide attention from the computational linguistics community.Many prior studies have shown that ChatGPT achieves remarkable performance on various NLP tasks in terms of automatic evaluation metrics.However, the ability of ChatGPT to serve as an evaluation metric is still underexplored.Considering assessing the quality of natural language generation (NLG) models is an arduous task and NLG metrics notoriously show their poor correlation with human judgments, we wonder whether Chat-GPT is a good NLG evaluation metric.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This preliminary meta-evaluation prompts ChatGPT as an NLG evaluator with task- and aspect-specific instructions and tests its correlation with human judgments on five meta-evaluation datasets. It finds ChatGPT achieves state-of-the-art or competitive correlations in most cases, but its effectiveness is prompt-sensitive and can be undermined by reference-oriented dataset biases.
Key findings
- On SummEval, ChatGPT achieves new state-of-the-art correlation in most aspects; ChatGPT (Stars w/ ref) reaches average sample-level Spearman 0.411 (Table 1).
- On OpenMEVA story generation, ChatGPT achieves the best performance; ChatGPT (DA w/o ref) reaches 0.507 sample-level Spearman correlation, while PPL reaches only 0.324.
- On RealSumm, ChatGPT underperforms similarity metrics; ChatGPT (Stars w/ ref) reaches 0.195 sample-level Spearman correlation versus ROUGE-1 at 0.498, which the paper attributes to reference-oriented pyramid annotation bias.
- On NewsRoom, ChatGPT outperforms ROUGE and BERTScore by a large margin; ChatGPT (DA w/o ref) reaches average sample-level Spearman 0.504 versus ROUGE-1 at 0.119 and BERTScore at 0.152. On BAGEL, ChatGPT obtains competitive correlations.
Methods
- Meta-evaluation on five NLG meta-evaluation datasets: SummEval, NewsRoom, and RealSumm for summarization; OpenMEVA-ROC for story generation; BAGEL for data-to-text generation.
- ChatGPT is prompted as a human evaluator with task-specific and aspect-specific instructions, using reference-free and reference-based settings, and direct assessment (0–100) or one-to-five stars prompts.
- Compared against ROUGE-1/2/L, BERTScore, MoverScore, PRISM, BARTScore variants, and PPL; correlations are measured with Spearman, Pearson, and Kendall’s Tau at sample and dataset levels.
Limitations
- Stated: ChatGPT’s performance is prompt-sensitive; experiments miss mainstream NLG tasks such as dialogue and report generation; no official API was available, so experiments used the ChatGPT website with default temperature and are difficult to reproduce; only English datasets are used; correlation depends on the quality and challenge of the meta-evaluation datasets, which are traditional pre-LLM datasets.
- Evident from the text: this is a preliminary report with limited task and dataset coverage, and the lack of an official API/default temperature implies non-deterministic, version-dependent results.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
17 Own result · 10 Own interpretation · 6 Not checked yet · 5 No basis given · 4 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we also find that all similarity-based metrics (i.e., ROUGE-1, ROUGE-2, ROUGE-L, BERTScore and BARTScore) show their weak correlations with the human judgments.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 7 shows the experimental results, where ChatGPT achieves competitive correlations compared with the previous state-of-the-art baselines, indicating its strong potentiality serving as a metric for data-to-text generation.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.5 Data-to-Text Generation
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Besides, the performances of ChatGPT as an NLG evaluator are sensitive to the format of the prompt, for different tasks and aspects, the prompt should be carefully designed.”
- Also stated in: § Limitations · plain
“(1) Chat-GPT's performance as an NLG metric is related to prompts, and future work could explore more powerful prompts to achieve better performance;”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 5 Conclusion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
- Also stated in: § Limitations · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we find that the effectiveness of the ChatGPT evaluator might be influenced by the creation method of the meta-evaluation datasets.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Abstract
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results show that ChatGPT achieves a new state-of-the-art correlation in most aspects of SummEval, demonstrating its potential of serving as an NLG metric.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For results on Newsroom, ChatGPT also outperforms dominant summarization metrics (i.e., ROUGE and BERTScore) by a large margin.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Moreover, the creation method of the meta-evaluation datasets has a significant influence on the effectiveness of different evaluation metrics.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results in Table 6 also show the sensitivity of the correlation results led by the different prompts.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > The Impact of Different Prompt.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results show that compared with previous automatic metrics, ChatGPT achieves state-of-the-art or competitive correlation with human judgments in most cases.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT achieves the new state-of-theart correlations (with human judgments) on Sum-mEval and OpenMEVA meta-evaluation datasets, and obtains competitive results on NewsRoom and BAGEL datasets.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 5 Conclusion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT achieves the new state-of-theart correlations (with human judgments) on Sum-mEval and OpenMEVA meta-evaluation datasets, and obtains competitive results on NewsRoom and BAGEL datasets.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 5 Conclusion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As we can see, the performances of ChatGPT are sensitive to the prompt design.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, there are large performance gaps between ChatGPT (DA w/o ref) and ChatGPT (DA w/ ref).”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > The Impact of Different Prompt.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“indicating its potential as an NLG metric.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 7 shows the experimental results, where ChatGPT achieves competitive correlations compared with the previous state-of-the-art baselines, indicating its strong potentiality serving as a metric for data-to-text generation.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.5 Data-to-Text Generation
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this manner, the more similarity between one generated summary and the corresponding golden reference, the more human evaluation scores will be achieved.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the meta-evaluation datasets which are created greatly depending on the reference and thus are biased, the ChatGPT evaluator might lose its effectiveness.”
- Also stated in: § 1 Introduction · hedged
“Therefore, the ChatGPT evaluator might lose its effectiveness in such situations.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Abstract
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · hedged
In the paper
“We conjecture this is because the human judgments in RealSumm are collected via pyramid method (Nenkova and Passonneau, 2004).”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
What the sources say
- Not checked yetcites Nenkova and Passonneau, 2004
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This finding indicates that the ChatGPT evaluator has more powerful and reliable judgments on the open-ended and creative text generation tasks, where many diversified generated results could also be regarded as high-quality.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results on five widely-used meta-evaluation datasets, covering text summarization, story generation and data-to-text tasks, show the potentiality of ChatGPT as an NLG metric.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 5 Conclusion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
- Rests on an assumption: The task-specific and aspect-specific prompts used are adequate to elicit ChatGPT's NLG evaluation capability.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The automatic similarity-based metrics between the generated storylines and so-called references cannot fully evaluate the quality of the storylines since they do not consider creativity.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 12
- It may state two separate claims as one. 12
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This finding is also consistent with that in text summarization (Section 4.3).”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > The Impact of Different Prompt.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that the ChatGPT evaluator has a high correlation with humans in most cases, especially for creative NLG tasks (e.g., story generation) where multiple generations can satisfy humans.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In addition, we also find that the lexical biases involved in the meta-evaluation datasets would influence the effectiveness of NLG metrics, and might lead to the limited performance of the ChatGPT evaluator.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 5 Conclusion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, ChatGPT (DA w/o ref) achieves 0.507 sample-level Spearman correlation, while PPL only achieves 0.324 sample-level Spearman correlation.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 6, ChatGPT achieves the best performance in terms of all correlations, and significantly outperforms the second-best metric (i.e., PPL).”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, ChatGPT (Stars w/ ref) achieves 0.195 sample-level Spearman correlation, which is far behind the counterpart of ROUGE-1 (i.e., 0.498).”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We find that the involved biases of the NLG metaevaluation datasets also influence the effectiveness of NLG metrics, and might lead to the limited effectiveness of the ChatGPT evaluator.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results show that ChatGPT exhibits a high correlation with human judgment in most cases especially for the story generation task, indicating its potential as an NLG metric.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“(3) When we did the experiments, the OpenAI ChatGPT did not release the official API.Thus, we conducted the experiments on the ChatGPT website with default temperature, making the results difficult to reproduce.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiments are conducted on the traditional NLG meta-evaluation datasets (that appear before the LLM era).”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“(4) The experiments are only conducted on the English NLG meta-evaluation datasets, and future work could extend this method into other languages or cross-lingual scenes.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Newsroom, as a text summarization dataset, also provides human judgments on 7 model-generated summaries, including coherence, relevance, informativeness and fluency.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
What the sources say
- Not checked yetcites Grusky et al., 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, this reference-oriented annotation method makes the traditional n-grambased metric (such as ROUGE) already achieve well correlations with human judgments, which we named as lexical biases.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In detail, this method first requires human evaluators to extract semantic content units from golden references, and then score each system summary based on how many extracted semantic content units are mentioned in the system summary.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
What the sources say
- Not checked yetcites Nenkova and Passonneau, 2004
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(5) The correlation between the ChatGPT evaluator and humans is also related to the quality and challenge of the corresponding meta-evaluation datasets.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
In the paper
“The OpenMEVA-ROC dataset manually annotates five model-generated storylines under the consideration of their overall quality.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
What the sources say
- Not checked yetcites Guan et al., 2021
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If a meta-evaluation dataset is created greatly depending on the reference, the similarity between model generation and references serves as a strong signal to reflect human judgments, where simple similarity-based metrics (e.g., ROUGE) can achieve very strong performance.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that our experiments only estimate the lower bound of ChatGPT's performance, and better performances would like to be achieved once using better prompts or updated versions of ChatGPT.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“(2) This preliminary report misses experiments on some mainstream NLG tasks, e.g., dialogue generation and report generation;”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“SummEval collects 16 modelgenerated summaries on the CNN/DM dataset and annotates human judgments upon these summaries covering aspects of coherence, relevance, consistency and fluency.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
What the sources say
- Not checked yetcites Fabbri et al., 2021
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“RealSumm evaluates the pyramid (Nenkova and Passonneau, 2004) recall of 25 model-generated summaries.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
What the sources say
- Not checked yetcites Bhandari et al., 2020
- Not checked yetcites Nenkova and Passonneau, 2004
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The ability of ChatGPT to serve as an evaluation metric is still underexplored.
“However, the ability of ChatGPT to serve as an evaluation metric is still underexplored.”
The paper relies on this without support.
The automated assessment of the general generation quality of NLG models still remains underexplored.
“However, the automated assessment of the general generation quality of NLG models still remains underexplored.”
The paper relies on this without support.
The human judgments in the meta-evaluation datasets are reliable enough to serve as ground truth for evaluating NLG metrics.
“SummEval collects 16 modelgenerated summaries on the CNN/DM dataset and annotates human judgments upon these summaries covering aspects of coherence, relevance, consistency and fluency.”
The paper relies on this without support.
The five meta-evaluation datasets are representative of NLG tasks and suitable for evaluating general NLG metrics.
“We conduct experiments on five NLG metaevaluation datasets (including summarization, story generation and data-to-text tasks).”
The paper relies on this without support.
For a given beginning of a story, various generated storylines and different plots could satisfy people.
“for a given beginning of a story, various generated storylines and different plots could satisfy people.”
The paper relies on this without support.
Human judgments are a valid gold standard for evaluating NLG metrics.
“ChatGPT achieves the new state-of-theart correlations (with human judgments) on Sum-mEval and OpenMEVA meta-evaluation datasets”
The paper relies on this without support.
The task-specific and aspect-specific prompts used are adequate to elicit ChatGPT's NLG evaluation capability.
“To this end, we design task-specific as well as aspectspecific prompts to guide ChatGPT to perform as an NLG metric.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GPT-3.5 has been proved to have positive correlation with human annotations (Chiang and Lee, 2023a;Wang et al., 2023a)”
From Bayesian Calibration of Win Rate Estimation with LLM Evaluators (2024) · cites this paper as Wang et al., 2023a
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recent studies have demonstrated that large language models (LLMs) can serve as reliable evaluators for natural language generation (NLG) tasks, exhibiting strong correlation with human judgments.”
From Enhancing memory retrieval in generative agents through LLM-trained cross attention networks (2025) · cites this paper as Wang et al. (2023)
In the citing paper
“Some other evaluators based on LLMs [37,60,74,118] also show good human alignment in more NLG tasks, especially compared with traditional metrics.”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [118]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
“ChatGPT (DA w/o ref) achieves 0.507 sample-level Spearman correlation, while PPL only achieves 0.324 sample-level Spearman correlation.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
“| ChatGPT (DA w/o ref) ChatGPT (Stars w/o ref) 0.472 0.490 0.427 0.415 0.439 0.342 0.507 0.533 0.439 0.471 0.494 0.366 | |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| ROUGE-2 | 0.026 -0.108 0.009 0.091 -0.056 0.065 0.079 -0.087 0.052 0.048 -0.101 0.032 0.061 -0.088 0.092 |”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
Reported 0.092 · recomputed 0.0395
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.1 Experimental Setup
“Two widely-used correlation measures are adopted:”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.1 Experimental Setup
“(3) Kendall's Tau (Kendall, 1938) evaluates the ordinal association between two measured quantities.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
(2004) · Cited by 9,718
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
A NEW MEASURE OF RANK CORRELATION
Kendall, M. G. (1938) · Biometrika · doi:10.1093/biomet/30.1-2.81 · Cited by 7,299
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 2,459
A New Measure of Rank Correlation
Kendall, M. G. (1938) · Biometrika · doi:10.2307/2332226 · Cited by 2,346
From Word Embeddings To Document Distances
Kusner, M. J., Sun, Y., Kolkin, N., & Weinberger, K. Q. (2015) · PolyPublie (École Polytechnique de Montréal) · Cited by 1,695
Research works printed but not linked: 3
- Mavuto M Mukaka. 2012. A guide to appropriate use of correlation coefficient in medical research. Malawi medical journal, 24(3):69-71.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computa- tional Linguistics, ACL '02, page 311-318, USA. Association for Computational Linguistics.
- Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023a. Cross-lingual summarization via chatgpt. arXiv preprint.
Cited by
Sorted by citations
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., & Liu, Z. (2023) · arXiv · doi:10.48550/arxiv.2308.07201 · Cited by 725
Metadata
- Authors
- Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J.
- Affiliations
- Soochow University (CN)
- Beijing Jiaotong University (CN)
- Tencent (China) (CN)
- Waseda University (JP)
- Fudan University (CN)
- Venue
- Proceedings of the 4th New Frontiers in Summarization Workshop
- Published
- 2023
- Pages
- 1-11
- OpenAlex
- W4389519239
- Cited by
- 235
- References
- 42
- Citations by year
- 2026: 43 · 2025: 65 · 2024: 75 · 2023: 52
- FWCI
- 41.64
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Workshop on New Frontiers in Summarization (NewSum) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- ChatGPT, natural language generation, automatic evaluation metrics, large language models, human judgment correlation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
