The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; none of the 7 sources its claims cite is checked for support yet.
Is ChatGPT a Good NLG Evaluator? A Preliminary Study
Open accessAbstract
Recently, the emergence of ChatGPT has attracted wide attention from the computational linguistics community.Many prior studies have shown that ChatGPT achieves remarkable performance on various NLP tasks in terms of automatic evaluation metrics.However, the ability of ChatGPT to serve as an evaluation metric is still underexplored.Considering assessing the quality of natural language generation (NLG) models is an arduous task and NLG metrics notoriously show their poor correlation with human judgments, we wonder whether Chat-GPT is a good NLG evaluation metric.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This preliminary meta-evaluation prompts ChatGPT as an NLG evaluator with task- and aspect-specific instructions and tests its correlation with human judgments on five meta-evaluation datasets. It finds ChatGPT achieves state-of-the-art or competitive correlations in most cases, but its effectiveness is prompt-sensitive and can be undermined by reference-oriented dataset biases.
Key findings
- On SummEval, ChatGPT achieves new state-of-the-art correlation in most aspects; ChatGPT (Stars w/ ref) reaches average sample-level Spearman 0.411 (Table 1).
- On OpenMEVA story generation, ChatGPT achieves the best performance; ChatGPT (DA w/o ref) reaches 0.507 sample-level Spearman correlation, while PPL reaches only 0.324.
- On RealSumm, ChatGPT underperforms similarity metrics; ChatGPT (Stars w/ ref) reaches 0.195 sample-level Spearman correlation versus ROUGE-1 at 0.498, which the paper attributes to reference-oriented pyramid annotation bias.
- On NewsRoom, ChatGPT outperforms ROUGE and BERTScore by a large margin; ChatGPT (DA w/o ref) reaches average sample-level Spearman 0.504 versus ROUGE-1 at 0.119 and BERTScore at 0.152. On BAGEL, ChatGPT obtains competitive correlations.
Methods
- Meta-evaluation on five NLG meta-evaluation datasets: SummEval, NewsRoom, and RealSumm for summarization; OpenMEVA-ROC for story generation; BAGEL for data-to-text generation.
- ChatGPT is prompted as a human evaluator with task-specific and aspect-specific instructions, using reference-free and reference-based settings, and direct assessment (0–100) or one-to-five stars prompts.
- Compared against ROUGE-1/2/L, BERTScore, MoverScore, PRISM, BARTScore variants, and PPL; correlations are measured with Spearman, Pearson, and Kendall’s Tau at sample and dataset levels.
Limitations
- Stated: ChatGPT’s performance is prompt-sensitive; experiments miss mainstream NLG tasks such as dialogue and report generation; no official API was available, so experiments used the ChatGPT website with default temperature and are difficult to reproduce; only English datasets are used; correlation depends on the quality and challenge of the meta-evaluation datasets, which are traditional pre-LLM datasets.
- Evident from the text: this is a preliminary report with limited task and dataset coverage, and the lack of an official API/default temperature implies non-deterministic, version-dependent results.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
17 Own result · 10 Own interpretation · 6 Not checked yet · 5 No basis given · 4 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we also find that all similarity-based metrics (i.e., ROUGE-1, ROUGE-2, ROUGE-L, BERTScore and BARTScore) show their weak correlations with the human judgments.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 7 shows the experimental results, where ChatGPT achieves competitive correlations compared with the previous state-of-the-art baselines, indicating its strong potentiality serving as a metric for data-to-text generation.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.5 Data-to-Text Generation
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Besides, the performances of ChatGPT as an NLG evaluator are sensitive to the format of the prompt, for different tasks and aspects, the prompt should be carefully designed.”
- Also stated in: § Limitations · plain
“(1) Chat-GPT's performance as an NLG metric is related to prompts, and future work could explore more powerful prompts to achieve better performance;”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 5 Conclusion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
- Also stated in: § Limitations · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we find that the effectiveness of the ChatGPT evaluator might be influenced by the creation method of the meta-evaluation datasets.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Abstract
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results show that ChatGPT achieves a new state-of-the-art correlation in most aspects of SummEval, demonstrating its potential of serving as an NLG metric.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For results on Newsroom, ChatGPT also outperforms dominant summarization metrics (i.e., ROUGE and BERTScore) by a large margin.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Moreover, the creation method of the meta-evaluation datasets has a significant influence on the effectiveness of different evaluation metrics.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results in Table 6 also show the sensitivity of the correlation results led by the different prompts.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > The Impact of Different Prompt.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results show that compared with previous automatic metrics, ChatGPT achieves state-of-the-art or competitive correlation with human judgments in most cases.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT achieves the new state-of-theart correlations (with human judgments) on Sum-mEval and OpenMEVA meta-evaluation datasets, and obtains competitive results on NewsRoom and BAGEL datasets.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 5 Conclusion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
- Probably not a substantial claim. 12
- It may state two separate claims as one. 12
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This finding is also consistent with that in text summarization (Section 4.3).”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > The Impact of Different Prompt.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that the ChatGPT evaluator has a high correlation with humans in most cases, especially for creative NLG tasks (e.g., story generation) where multiple generations can satisfy humans.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In addition, we also find that the lexical biases involved in the meta-evaluation datasets would influence the effectiveness of NLG metrics, and might lead to the limited performance of the ChatGPT evaluator.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 5 Conclusion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, ChatGPT (DA w/o ref) achieves 0.507 sample-level Spearman correlation, while PPL only achieves 0.324 sample-level Spearman correlation.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 6, ChatGPT achieves the best performance in terms of all correlations, and significantly outperforms the second-best metric (i.e., PPL).”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, ChatGPT (Stars w/ ref) achieves 0.195 sample-level Spearman correlation, which is far behind the counterpart of ROUGE-1 (i.e., 0.498).”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We find that the involved biases of the NLG metaevaluation datasets also influence the effectiveness of NLG metrics, and might lead to the limited effectiveness of the ChatGPT evaluator.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results show that ChatGPT exhibits a high correlation with human judgment in most cases especially for the story generation task, indicating its potential as an NLG metric.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“(3) When we did the experiments, the OpenAI ChatGPT did not release the official API.Thus, we conducted the experiments on the ChatGPT website with default temperature, making the results difficult to reproduce.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiments are conducted on the traditional NLG meta-evaluation datasets (that appear before the LLM era).”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“(4) The experiments are only conducted on the English NLG meta-evaluation datasets, and future work could extend this method into other languages or cross-lingual scenes.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Newsroom, as a text summarization dataset, also provides human judgments on 7 model-generated summaries, including coherence, relevance, informativeness and fluency.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
What the sources say
- Not checked yetcites Grusky et al., 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, this reference-oriented annotation method makes the traditional n-grambased metric (such as ROUGE) already achieve well correlations with human judgments, which we named as lexical biases.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In detail, this method first requires human evaluators to extract semantic content units from golden references, and then score each system summary based on how many extracted semantic content units are mentioned in the system summary.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
What the sources say
- Not checked yetcites Nenkova and Passonneau, 2004
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(5) The correlation between the ChatGPT evaluator and humans is also related to the quality and challenge of the corresponding meta-evaluation datasets.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
In the paper
“The OpenMEVA-ROC dataset manually annotates five model-generated storylines under the consideration of their overall quality.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
What the sources say
- Not checked yetcites Guan et al., 2021
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If a meta-evaluation dataset is created greatly depending on the reference, the similarity between model generation and references serves as a strong signal to reflect human judgments, where simple similarity-based metrics (e.g., ROUGE) can achieve very strong performance.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that our experiments only estimate the lower bound of ChatGPT's performance, and better performances would like to be achieved once using better prompts or updated versions of ChatGPT.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“(2) This preliminary report misses experiments on some mainstream NLG tasks, e.g., dialogue generation and report generation;”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“SummEval collects 16 modelgenerated summaries on the CNN/DM dataset and annotates human judgments upon these summaries covering aspects of coherence, relevance, consistency and fluency.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
What the sources say
- Not checked yetcites Fabbri et al., 2021
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“RealSumm evaluates the pyramid (Nenkova and Passonneau, 2004) recall of 25 model-generated summaries.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
What the sources say
- Not checked yetcites Bhandari et al., 2020
- Not checked yetcites Nenkova and Passonneau, 2004
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The ability of ChatGPT to serve as an evaluation metric is still underexplored.
“However, the ability of ChatGPT to serve as an evaluation metric is still underexplored.”
The paper relies on this without support.
The automated assessment of the general generation quality of NLG models still remains underexplored.
“However, the automated assessment of the general generation quality of NLG models still remains underexplored.”
The paper relies on this without support.
The human judgments in the meta-evaluation datasets are reliable enough to serve as ground truth for evaluating NLG metrics.
“SummEval collects 16 modelgenerated summaries on the CNN/DM dataset and annotates human judgments upon these summaries covering aspects of coherence, relevance, consistency and fluency.”
The paper relies on this without support.
The five meta-evaluation datasets are representative of NLG tasks and suitable for evaluating general NLG metrics.
“We conduct experiments on five NLG metaevaluation datasets (including summarization, story generation and data-to-text tasks).”
The paper relies on this without support.
For a given beginning of a story, various generated storylines and different plots could satisfy people.
“for a given beginning of a story, various generated storylines and different plots could satisfy people.”
The paper relies on this without support.
Human judgments are a valid gold standard for evaluating NLG metrics.
“ChatGPT achieves the new state-of-theart correlations (with human judgments) on Sum-mEval and OpenMEVA meta-evaluation datasets”
The paper relies on this without support.
The task-specific and aspect-specific prompts used are adequate to elicit ChatGPT's NLG evaluation capability.
“To this end, we design task-specific as well as aspectspecific prompts to guide ChatGPT to perform as an NLG metric.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GPT-3.5 has been proved to have positive correlation with human annotations (Chiang and Lee, 2023a;Wang et al., 2023a)”
From Bayesian Calibration of Win Rate Estimation with LLM Evaluators (2024) · cites this paper as Wang et al., 2023a
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recent studies have demonstrated that large language models (LLMs) can serve as reliable evaluators for natural language generation (NLG) tasks, exhibiting strong correlation with human judgments.”
From Enhancing memory retrieval in generative agents through LLM-trained cross attention networks (2025) · cites this paper as Wang et al. (2023)
In the citing paper
“Some other evaluators based on LLMs [37,60,74,118] also show good human alignment in more NLG tasks, especially compared with traditional metrics.”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [118]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
“ChatGPT (DA w/o ref) achieves 0.507 sample-level Spearman correlation, while PPL only achieves 0.324 sample-level Spearman correlation.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
“| ChatGPT (DA w/o ref) ChatGPT (Stars w/o ref) 0.472 0.490 0.427 0.415 0.439 0.342 0.507 0.533 0.439 0.471 0.494 0.366 | |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| ROUGE-2 | 0.026 -0.108 0.009 0.091 -0.056 0.065 0.079 -0.087 0.052 0.048 -0.101 0.032 0.061 -0.088 0.092 |”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
Reported 0.092 · recomputed 0.0395
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.1 Experimental Setup
“Two widely-used correlation measures are adopted:”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.1 Experimental Setup
“(3) Kendall's Tau (Kendall, 1938) evaluates the ordinal association between two measured quantities.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
(2004) · Cited by 9,718
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
A NEW MEASURE OF RANK CORRELATION
Kendall, M. G. (1938) · Biometrika · doi:10.1093/biomet/30.1-2.81 · Cited by 7,299
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 2,459
A New Measure of Rank Correlation
Kendall, M. G. (1938) · Biometrika · doi:10.2307/2332226 · Cited by 2,346
From Word Embeddings To Document Distances
Kusner, M. J., Sun, Y., Kolkin, N., & Weinberger, K. Q. (2015) · PolyPublie (École Polytechnique de Montréal) · Cited by 1,695
Research works printed but not linked: 3
- Mavuto M Mukaka. 2012. A guide to appropriate use of correlation coefficient in medical research. Malawi medical journal, 24(3):69-71.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computa- tional Linguistics, ACL '02, page 311-318, USA. Association for Computational Linguistics.
- Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023a. Cross-lingual summarization via chatgpt. arXiv preprint.
Cited by
Sorted by citations
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.153 · Cited by 948
G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023) · arXiv · doi:10.48550/arxiv.2303.16634 · Cited by 948
Summary of ChatGPT-Related research and perspective towards the future of large language models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100017 · Cited by 855
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., & Liu, Z. (2023) · arXiv · doi:10.48550/arxiv.2308.07201 · Cited by 725
Harnessing the power of llms in practice: a survey on chatGPT and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649506 · Cited by 644
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Can Large Language Models Be an Alternative to Human Evaluations?
Chiang, C.-H., & Lee, H.-Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.870 · Cited by 331
Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving
Chen, L., Sinavski, O., Hünermann, J., Karnsund, A., Willmott, A. J., Birch, D., Maund, D., & Shotton, J. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611018 · Cited by 287
ChatAgri: Exploring potentials of ChatGPT on cross-linguistic agricultural text classification
Zhao, B., Jin, W., Ser, J. D., & Yang, G. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126708 · Cited by 156
A comprehensive survey of large language models and multimodal large language models in medicine
Xiao, H., Zhou, F., Liu, X., Liu, T., Ли, Ж., Liu, X., & Huang, X. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102888 · Cited by 154
Large Language Models for Networking: Applications, Enabling Techniques, and Challenges
Huang, Y., Du, H., Zhang, X., Niyato, D. T., Kang, J., Xiong, Z., Wang, S., & Huang, T. (2024) · IEEE Network · doi:10.1109/mnet.2024.3435752 · Cited by 108
LLM-based NLG Evaluation: Current Status and Challenges
Gao, M., Hu, X., Yin, X., Ruan, J., Pu, X., & Wan, X. (2025) · Computational Linguistics · doi:10.1162/coli_a_00561 · Cited by 243
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
Laskar, M. T. R., Bari, M. S., Rahman, M., Bhuiyan, M. A. H., Joty, S., & Huang, J. X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.29 · Cited by 160
Cheng, L., Li, X., & Bing, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.637 · Cited by 100
Scaria, N., Chenna, S. D., & Subramani, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-64299-9_12 · Cited by 96
Benchmarking Cognitive Biases in Large Language Models as Evaluators
Koo, R., Lee, M., Raheja, V., Park, J. I., Kim, Z. M., & Kang, D. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.29 · Cited by 70
Chen, Y., Wang, R., Jiang, H., Shi, S., & Xu, R. (2023) · doi:10.18653/v1/2023.findings-ijcnlp.32 · Cited by 142
LLM4Decompile: Decompiling Binary Code with Large Language Models
Tan, H., Luo, Q., Li, J., & Zhang, Y. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.203 · Cited by 67
GeneAgent: self-verification language agent for gene-set analysis using domain databases
Wang, Z., Jin, Q., Wei, C.-H., Tian, S., Lai, P., Zhu, Q., Day, C., Ross, C., Leaman, R., & Lu, Z. (2025) · Nature Methods · doi:10.1038/s41592-025-02748-6 · Cited by 114
Source Code Summarization in the Era of Large Language Models
Sun, W., Miao, Y., Li, Y., Zhang, H., Fang, C., Liu, Y., Deng, G., Liu, Y., & Chen, Z. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00034 · Cited by 157
Farrukh, Y. A., Wali, S. W., Khan, I., & Bastian, N. D. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.128089 · Cited by 62
Jatmo: Prompt Injection Defense by Task-Specific Finetuning
Piet, J., Al-Rashed, M., Sitawarin, C., Chen, S., Wei, Z., Sun, E., Alomair, B., & Wagner, D. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70879-4_6 · Cited by 124
Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization
Shen, C., Cheng, L., Nguyen, X.-P., You, Y., & Bing, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.278 · Cited by 112
Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors
Tang, L., Goyal, T., Fabbri, A. R., Laban, P., Xu, J., Yavuz, S., Kryściński, W., Rousseau, J. F., & Durrett, G. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.650 · Cited by 87
Self-citationLi, M., Shi, T., Ziems, C., Kan, M., Chen, N. F., Liu, Z., & Yang, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.92 · Cited by 92
Zambrano Chaves, J. M., Huang, S.-C., Xu, Y., Xu, H., Usuyama, N., Zhang, S., Wang, F., Xie, Y., Khademi, M., Yang, Z., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-58344-x · Cited by 45
Factual consistency evaluation of summarization in the Era of large language models
Luo, Z., Xie, Q., & Ananiadou, S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124456 · Cited by 59
Could ChatGPT get an engineering degree? Evaluating higher education vulnerability to AI assistants
Borges, B., Foroutan, N., Bayazit, D., Sotnikova, A., Montariol, S., Nazaretsky, T., Banaei, M., Sakhaeirad, A., Servant, P., Neshaei, S. P., et al. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2414955121 · Cited by 56
Zero-Shot Cross-Lingual Summarization via Large Language Models
Wang, J., Liang, Y., Meng, F., Zou, B., Li, Z., Qu, J., & Zhou, J. (2023) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2023.newsum-1.2 · Cited by 81
Self-citationCue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMs
Wang, H., Wang, R., Mi, F., Deng, Y., Wang, Z., Liang, B., Xu, R., & Wong, K.-F. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.806 · Cited by 38
Huang, X., Liu, Z. Y., Zhang, M., 桂, 韬., 韩, 先., 户, 保., 黄, 民., 黄, 萱., 刘, 康., 刘, 挺., et al. (2023) · Scientia Sinica Informationis · doi:10.1360/ssi-2023-0113 · Cited by 42
Cheng, S., Wen, J., Kang, J., Wang, Y., Su, Y., Pan, H., Zhong, Z., & Hossain, M. S. (2024) · IEEE Internet of Things Journal · doi:10.1109/jiot.2024.3521425 · Cited by 40
G-Refer: Graph Retrieval-Augmented Large Language Model for Explainable Recommendation
Li, Y., Zhang, X., Luo, L., Chang, H., Ren, Y., King, I., & Li, J. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714727 · Cited by 52
Tang, T., Sha, J., Zhao, Y., Wang, S., Wang, Z., & Shen, S. (2024) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2024.101607 · Cited by 52
Reinforcement Learning With LLMs Interaction for Distributed Diffusion Model Services
Du, H.-Y., Zhang, R.-C., Niyato, D., Kang, J., Xiong, Z.-H., Cui, S.-G., Shen, X.-M., & Kim, D. I. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3584698 · Cited by 45
Large Language Models as Evaluators for Recommendation Explanations
Zhang, X., Li, Y., Wang, J., Sun, B., Ma, W., Sun, P., & Zhang, M. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688075 · Cited by 47
Self-citationEvaluating zero-shot multilingual aspect-based sentiment analysis with large language models
Wu, C., Ma, B., Zhang, Z., Deng, N., He, Y., & Xue, Y. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-025-02711-z · Cited by 41
Nasseri, M., Brandtner, P., Zimmermann, R., Falatouri, T. N., Darbanian, F., & Obinwanne, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-48057-7_12 · Cited by 31
Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
Hua, A., Tang, K., Gu, C., Gu, J., Wong, E., & Qin, Y. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1006 · Cited by 28
Gursesli, M. C., Taveekitworachai, P., Abdullah, F., Dewantoro, M. F., Lanatà, A., Guazzini, A., Lê, V. K., Villars, A., & Thawonmas, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47658-7_16 · Cited by 29
Alomari, E. A. (2024) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2024.052256 · Cited by 37
LLM supervised Pre-training for Multimodal Emotion Recognition in Conversations
Dutta, S., & Ganapathy, S. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889998 · Cited by 38
Unveiling the Implicit Toxicity in Large Language Models
Wen, J., Ke, P., Sun, H., Zhang, Z., Li, C., Bai, J., & Huang, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.84 · Cited by 76
Lai, H., & Nissim, M. (2024) · ACM Computing Surveys · doi:10.1145/3654795 · Cited by 32
MediGPT: Exploring Potentials of Conventional and Large Language Models on Medical Data
Rony, M. A. T., Islam, M. S., Sultan, T., AlShathri, S. I., & El‐Shafai, W. (2024) · IEEE Access · doi:10.1109/access.2024.3428918 · Cited by 26
Fu, L., & Liu, L. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03726-7 · Cited by 24
An Evaluation of Reasoning Capabilities of Large Language Models in Financial Sentiment Analysis
Du, K., Xing, F., Mao, R., & Cambria, E. (2024) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai59869.2024.00042 · Cited by 24
Prompt-Gaming: A Pilot Study on LLM-Evaluating Agent in a Meaningful Energy Game
Isaza-Giraldo, A., Bala, P., Campos, P., & Pereira, L. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650774 · Cited by 21
GPT-4 as an X data annotator: Unraveling its performance on a stance classification task
Liyanage, C. R., Gokani, R., & Mago, V. (2024) · PLoS ONE · doi:10.1371/journal.pone.0307741 · Cited by 23
Banerjee, A., Satish, A., & Wörndl, W. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-87654-7_3 · Cited by 32
Tc-llama 2: fine-tuning LLM for technology and commercialization applications
Yeom, J., Lee, H., Byun, H., Kim, Y., Byun, J., Choi, Y., Kim, S., & Song, K. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00963-0 · Cited by 21
How Good Is ChatGPT in Giving Advice on Your Visualization Design?
Kim, N. W., Ahn, Y., Myers, G., & Bach, B. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3745768 · Cited by 45
Botunac, I., Bakarić, M. B., & Matetić, M. (2024) · Applied Sciences · doi:10.3390/app14146254 · Cited by 20
ArxivDIGESTables: Synthesizing Scientific Literature into Tables using Language Models
Newman, B., Lee, Y., Naik, A., Siangliulue, P., Fok, R., Kim, J., Weld, D. S., Chang, J. C., & Lo, K. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.538 · Cited by 19
Evaluation metrics on text summarization: comprehensive survey
Davoodijam, E., & Meybodi, M. A. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02217-0 · Cited by 23
Wysocka, M., Wysocki, O., Delmas, M., Mutel, V., & Freitas, A. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104724 · Cited by 33
A survey on large language models with multilingualism: recent advances and new frontiers
Huang, K., Mo, F., Zhang, X., Li, H., Li, Y., Zhang, Y., yi, W., Mao, Y., Liu, J., Xu, Y., et al. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11534-5 · Cited by 131
Self-citationZhang, Y., Xu, H., Zhang, D., & Xu, R. (2024) · Electronics · doi:10.3390/electronics13183724 · Cited by 21
Can ChatGPT Serve as a Multi-Criteria Decision Maker? A Novel Approach to Supplier Evaluation
Wang, X., & Wu, X. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447204 · Cited by 22
A Comparative Study of Sentiment Classification Models for Greek Reviews
Michailidis, P. D. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8090107 · Cited by 20
FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence
Joseph, S., Chen, L., Trienes, J., Göke, H., Coers, M., Xu, W., Wallace, B., & Li, J. J. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.459 · Cited by 17
Benayas, A., Sicilia, M., & Mora‐Cantallops, M. (2024) · Language Resources and Evaluation · doi:10.1007/s10579-024-09796-y · Cited by 22
What Is Waiting for Us at the End? Inherent Biases of Game Story Endings in Large Language Models
Taveekitworachai, P., Abdullah, F., Gursesli, M. C., Dewantoro, M. F., Chen, S., Lanatà, A., Guazzini, A., & Thawonmas, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47658-7_26 · Cited by 16
Explingo: Explaining AI Predictions using Large Language Models
Zytek, A., Pidò, S., Alnegheimish, S., Berti‐Équille, L., & Veeramachaneni, K. K. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825114 · Cited by 24
LLMEval: A Preliminary Study on How to Evaluate Large Language Models
Zhang, Y., Zhang, M., Yuan, H., Liu, S., Shi, Y., Gui, T., Zhang, Q., & Huang, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29934 · Cited by 31
Evidence, my Dear Watson: Abstractive dialogue summarization on learnable relevant utterances
Italiani, P., Frisoni, G., Moro, G., Carbonaro, A., & Sartori, C. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.127132 · Cited by 13
A Case Study on ChatGPT Question Generation
Chan, W., An, A., & Davoudi, H. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386520 · Cited by 15
Discrete Prompt Compression With Reinforcement Learning
Jung, H., & Kim, K.-J. (2024) · IEEE Access · doi:10.1109/access.2024.3403426 · Cited by 29
On Large Language Models’ Resilience to Coercive Interrogation
Zhang, Z., Shen, G., Tao, G., Cheng, S., & Zhang, X. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00208 · Cited by 20
FunQA: Towards Surprising Video Comprehension
Xie, B., Zhang, S., Zhou, Z., Li, B., Zhang, Y., Hessel, J., Yang, J., & Liu, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73232-4_3 · Cited by 13
Improving the Robustness of Knowledge-Grounded Dialogue via Contrastive Learning
Wang, J., Qu, J., Wang, K., Li, Z., Hua, W., Li, X., & Liu, A. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29881 · Cited by 12
Self-citationCan Large Language Models Serve as Evaluators for Code Summarization?
Wu, Y., Wan, Y., Chu, Z., Zhao, W., Liu, Y., Zhang, H., Shi, X., Jin, H., & Yu, P. S. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3595283 · Cited by 33
Proceedings of the 4th New Frontiers in Summarization Workshop
(2023) · doi:10.18653/v1/2023.newsum-1 · Cited by 12
Bhat, S., & Varma, V. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.8 · Cited by 12
Li, S., Chen, J., Yuan, S., Wu, X., Yang, H., Tao, S., & Xiao, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29817 · Cited by 32
Katzer, B., Klinder, S., & Schulz, K. (2025) · Materials Today Communications · doi:10.1016/j.mtcomm.2025.112186 · Cited by 15
Chu, S. Y., Kim, J. W., & Yi, M. Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713181 · Cited by 20
Evaluation of Question Answering Systems: Complexity of Judging a Natural Language
Farea, A., Yang, Z., Duong, K., Perera, N., & Emmert‐Streib, F. (2025) · ACM Computing Surveys · doi:10.1145/3744663 · Cited by 32
Liu, A., & Min, S. (2025) · AERA Open · doi:10.1177/23328584251374595 · Cited by 19
Breaking Bad: Unraveling Influences and Risks of User Inputs to ChatGPT for Game Story Generation
Taveekitworachai, P., Abdullah, F., Gursesli, M. C., Dewantoro, M. F., Chen, S., Lanatà, A., Guazzini, A., & Thawonmas, R. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47658-7_27 · Cited by 12
A Review on Optimization-Based Automatic Text Summarization Approach
Wahab, M. H. H., Ali, N. H., Hamid, N. A. W. A., Subramaniam, S. K., Latip, R. B., & Othman, М. (2023) · IEEE Access · doi:10.1109/access.2023.3348075 · Cited by 11
Sifting through the Chaff: On Utilizing Execution Feedback for Ranking the Generated Code Candidates
Sun, Z., Wan, Y., Li, J., Zhang, H., Jin, Z., Li, G., & Lyu, C. (2024) · Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/3691620.3695000 · Cited by 10
Self-citationJourney of ChatGPT from Prompts to Stories in Games: the Positive, the Negative, and the Neutral
Taveekitworachai, P., Gursesli, M. C., Abdullah, F., Chen, S., Calà, F., Guazzini, A., Lanatà, A., & Thawonmas, R. (2023) · IEEE International Conference on Consumer Electronics - Berlin (ICCE-Berlin) · doi:10.1109/icce-berlin58801.2023.10375663 · Cited by 10
A Novel Cascade Instruction Tuning Method for Biomedical NER
Zhao, J. Z., Liu, C., Liang, J., Li, Z., & Xiao, Y. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446885 · Cited by 11
Self-citationGopali, S., Siami‐Namini, S., Abri, F., & Namin, A. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100120 · Cited by 13
Senator, F., Lakhfif, A., Zenbout, I., Boutouta, H., & Mediani, C. (2025) · IEEE Access · doi:10.1109/access.2025.3525493 · Cited by 16
ChatGPT vs. Human Journalists: Analyzing News Summaries Through BERTScore and Moderation Standards
Kim, H.-S., Kang, J.-W., & Choi, S. (2025) · Electronics · doi:10.3390/electronics14112115 · Cited by 12
Tang, Z., Agarwal, M., Shypula, A., Wang, B., Wijaya, D. T., Chen, J., & Kim, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.119 · Cited by 19
Ferron, A., Shore, A., Mitra, E., & Agrawal, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.137 · Cited by 10
C5: toward better conversation comprehension and contextual continuity for ChatGPT
Liang, P., Ye, D., Zhu, Z., Wang, Y., Wang, X., Liang, R., & Sun, G. (2024) · Journal of Visualization · doi:10.1007/s12650-024-00980-4 · Cited by 11
Towards Efficient Educational Chatbots: Benchmarking RAG Frameworks
Khan, U. A., Khan, F., Khan, E., Hasnain, M. A., & Moinuddin, A. A. (2025) · International Conference on Emerging Technologies in Computing and Communication (ETCC) · doi:10.1109/etcc65847.2025.11108323 · Cited by 8
Exploring Dense Retrieval for Dialogue Response Selection
Lan, T., Cai, D., Wang, Y., Su, Y., Huang, H., & Mao, X.-L. (2023) · ACM Transactions on Information Systems · doi:10.1145/3632750 · Cited by 18
Governing Open Vocabulary Data Leaks Using an Edge LLM through Programming by Example
Li, Q., Wen, J., & Jin, H. (2024) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3699760 · Cited by 8
Is ChatGPT the Future of Causal Text Mining? A Comprehensive Evaluation and Analysis
Takayanagi, T., Suzuki, M., Kobayashi, R., Sakaji, H., & Izumi, K. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825555 · Cited by 9
Yoon, S.-H., Yang, S.-B., & Lee, S.-H. (2025) · Electronic Commerce Research and Applications · doi:10.1016/j.elerap.2025.101491 · Cited by 8
Empowering Social Service with AI: Insights from a Participatory Design Study with Practitioners
Tan, Y., Soh, K. X., Zhang, R., Lee, J., Meng, H., Sen, B., & Lee, Y. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719736 · Cited by 19
Grazhdanski, G., Vasilev, V., Vassileva, S., Taskov, D., Antova, I., Koychev, I. K., & Boytcheva, S. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104906 · Cited by 12
Enhancing Novelty in ChatGPT Responses: Incorporating Random Word Brainstorming
Taveekitworachai, P., & Thawonmas, R. (2023) · doi:10.1145/3628454.3628456 · Cited by 8
Su, X., Huang, Z., Su, Y., Trisedya, B. D., Dou, Y., & Zhao, Y. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3390223 · Cited by 10
Metadata
- Authors
- Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J.
- Affiliations
- Soochow University (CN)
- Beijing Jiaotong University (CN)
- Tencent (China) (CN)
- Waseda University (JP)
- Fudan University (CN)
- Venue
- Proceedings of the 4th New Frontiers in Summarization Workshop
- Published
- 2023
- Pages
- 1-11
- OpenAlex
- W4389519239
- Cited by
- 235
- References
- 42
- Citations by year
- 2026: 43 · 2025: 65 · 2024: 75 · 2023: 52
- FWCI
- 41.64
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Workshop on New Frontiers in Summarization (NewSum) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- ChatGPT, natural language generation, automatic evaluation metrics, large language models, human judgment correlation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
