The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index; none of the 7 sources its claims cite is checked for support yet.
Is ChatGPT a Good NLG Evaluator? A Preliminary Study
Open accessAbstract
Recently, the emergence of ChatGPT has attracted wide attention from the computational linguistics community.Many prior studies have shown that ChatGPT achieves remarkable performance on various NLP tasks in terms of automatic evaluation metrics.However, the ability of ChatGPT to serve as an evaluation metric is still underexplored.Considering assessing the quality of natural language generation (NLG) models is an arduous task and NLG metrics notoriously show their poor correlation with human judgments, we wonder whether Chat-GPT is a good NLG evaluation metric.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This preliminary meta-evaluation prompts ChatGPT as an NLG evaluator with task- and aspect-specific instructions and tests its correlation with human judgments on five meta-evaluation datasets. It finds ChatGPT achieves state-of-the-art or competitive correlations in most cases, but its effectiveness is prompt-sensitive and can be undermined by reference-oriented dataset biases.
Key findings
- On SummEval, ChatGPT achieves new state-of-the-art correlation in most aspects; ChatGPT (Stars w/ ref) reaches average sample-level Spearman 0.411 (Table 1).
- On OpenMEVA story generation, ChatGPT achieves the best performance; ChatGPT (DA w/o ref) reaches 0.507 sample-level Spearman correlation, while PPL reaches only 0.324.
- On RealSumm, ChatGPT underperforms similarity metrics; ChatGPT (Stars w/ ref) reaches 0.195 sample-level Spearman correlation versus ROUGE-1 at 0.498, which the paper attributes to reference-oriented pyramid annotation bias.
- On NewsRoom, ChatGPT outperforms ROUGE and BERTScore by a large margin; ChatGPT (DA w/o ref) reaches average sample-level Spearman 0.504 versus ROUGE-1 at 0.119 and BERTScore at 0.152. On BAGEL, ChatGPT obtains competitive correlations.
Methods
- Meta-evaluation on five NLG meta-evaluation datasets: SummEval, NewsRoom, and RealSumm for summarization; OpenMEVA-ROC for story generation; BAGEL for data-to-text generation.
- ChatGPT is prompted as a human evaluator with task-specific and aspect-specific instructions, using reference-free and reference-based settings, and direct assessment (0–100) or one-to-five stars prompts.
- Compared against ROUGE-1/2/L, BERTScore, MoverScore, PRISM, BARTScore variants, and PPL; correlations are measured with Spearman, Pearson, and Kendall’s Tau at sample and dataset levels.
Limitations
- Stated: ChatGPT’s performance is prompt-sensitive; experiments miss mainstream NLG tasks such as dialogue and report generation; no official API was available, so experiments used the ChatGPT website with default temperature and are difficult to reproduce; only English datasets are used; correlation depends on the quality and challenge of the meta-evaluation datasets, which are traditional pre-LLM datasets.
- Evident from the text: this is a preliminary report with limited task and dataset coverage, and the lack of an official API/default temperature implies non-deterministic, version-dependent results.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
17 Own result · 10 Own interpretation · 6 Not checked yet · 5 No basis given · 4 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we also find that all similarity-based metrics (i.e., ROUGE-1, ROUGE-2, ROUGE-L, BERTScore and BARTScore) show their weak correlations with the human judgments.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 7 shows the experimental results, where ChatGPT achieves competitive correlations compared with the previous state-of-the-art baselines, indicating its strong potentiality serving as a metric for data-to-text generation.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.5 Data-to-Text Generation
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Besides, the performances of ChatGPT as an NLG evaluator are sensitive to the format of the prompt, for different tasks and aspects, the prompt should be carefully designed.”
- Also stated in: § Limitations · plain
“(1) Chat-GPT's performance as an NLG metric is related to prompts, and future work could explore more powerful prompts to achieve better performance;”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 5 Conclusion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
- Also stated in: § Limitations · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, we find that the effectiveness of the ChatGPT evaluator might be influenced by the creation method of the meta-evaluation datasets.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Abstract
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results show that ChatGPT achieves a new state-of-the-art correlation in most aspects of SummEval, demonstrating its potential of serving as an NLG metric.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For results on Newsroom, ChatGPT also outperforms dominant summarization metrics (i.e., ROUGE and BERTScore) by a large margin.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Moreover, the creation method of the meta-evaluation datasets has a significant influence on the effectiveness of different evaluation metrics.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results in Table 6 also show the sensitivity of the correlation results led by the different prompts.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > The Impact of Different Prompt.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results show that compared with previous automatic metrics, ChatGPT achieves state-of-the-art or competitive correlation with human judgments in most cases.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Abstract
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT achieves the new state-of-theart correlations (with human judgments) on Sum-mEval and OpenMEVA meta-evaluation datasets, and obtains competitive results on NewsRoom and BAGEL datasets.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 5 Conclusion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
- Probably not a substantial claim. 12
- It may state two separate claims as one. 12
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This finding is also consistent with that in text summarization (Section 4.3).”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > The Impact of Different Prompt.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that the ChatGPT evaluator has a high correlation with humans in most cases, especially for creative NLG tasks (e.g., story generation) where multiple generations can satisfy humans.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In addition, we also find that the lexical biases involved in the meta-evaluation datasets would influence the effectiveness of NLG metrics, and might lead to the limited performance of the ChatGPT evaluator.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 5 Conclusion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, ChatGPT (DA w/o ref) achieves 0.507 sample-level Spearman correlation, while PPL only achieves 0.324 sample-level Spearman correlation.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Table 6, ChatGPT achieves the best performance in terms of all correlations, and significantly outperforms the second-best metric (i.e., PPL).”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, ChatGPT (Stars w/ ref) achieves 0.195 sample-level Spearman correlation, which is far behind the counterpart of ROUGE-1 (i.e., 0.498).”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We find that the involved biases of the NLG metaevaluation datasets also influence the effectiveness of NLG metrics, and might lead to the limited effectiveness of the ChatGPT evaluator.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Experimental results show that ChatGPT exhibits a high correlation with human judgment in most cases especially for the story generation task, indicating its potential as an NLG metric.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“(3) When we did the experiments, the OpenAI ChatGPT did not release the official API.Thus, we conducted the experiments on the ChatGPT website with default temperature, making the results difficult to reproduce.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our experiments are conducted on the traditional NLG meta-evaluation datasets (that appear before the LLM era).”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“(4) The experiments are only conducted on the English NLG meta-evaluation datasets, and future work could extend this method into other languages or cross-lingual scenes.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Newsroom, as a text summarization dataset, also provides human judgments on 7 model-generated summaries, including coherence, relevance, informativeness and fluency.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
What the sources say
- Not checked yetcites Grusky et al., 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, this reference-oriented annotation method makes the traditional n-grambased metric (such as ROUGE) already achieve well correlations with human judgments, which we named as lexical biases.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In detail, this method first requires human evaluators to extract semantic content units from golden references, and then score each system summary based on how many extracted semantic content units are mentioned in the system summary.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
What the sources say
- Not checked yetcites Nenkova and Passonneau, 2004
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“(5) The correlation between the ChatGPT evaluator and humans is also related to the quality and challenge of the corresponding meta-evaluation datasets.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: Human judgments are a valid gold standard for evaluating NLG metrics.
In the paper
“The OpenMEVA-ROC dataset manually annotates five model-generated storylines under the consideration of their overall quality.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
What the sources say
- Not checked yetcites Guan et al., 2021
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If a meta-evaluation dataset is created greatly depending on the reference, the similarity between model generation and references serves as a strong signal to reflect human judgments, where simple similarity-based metrics (e.g., ROUGE) can achieve very strong performance.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 1 Introduction
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that our experiments only estimate the lower bound of ChatGPT's performance, and better performances would like to be achieved once using better prompts or updated versions of ChatGPT.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“(2) This preliminary report misses experiments on some mainstream NLG tasks, e.g., dialogue generation and report generation;”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“SummEval collects 16 modelgenerated summaries on the CNN/DM dataset and annotates human judgments upon these summaries covering aspects of coherence, relevance, consistency and fluency.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
What the sources say
- Not checked yetcites Fabbri et al., 2021
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“RealSumm evaluates the pyramid (Nenkova and Passonneau, 2004) recall of 25 model-generated summaries.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.3 Text Summarization
What the sources say
- Not checked yetcites Bhandari et al., 2020
- Not checked yetcites Nenkova and Passonneau, 2004
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The ability of ChatGPT to serve as an evaluation metric is still underexplored.
“However, the ability of ChatGPT to serve as an evaluation metric is still underexplored.”
The paper relies on this without support.
The automated assessment of the general generation quality of NLG models still remains underexplored.
“However, the automated assessment of the general generation quality of NLG models still remains underexplored.”
The paper relies on this without support.
The human judgments in the meta-evaluation datasets are reliable enough to serve as ground truth for evaluating NLG metrics.
“SummEval collects 16 modelgenerated summaries on the CNN/DM dataset and annotates human judgments upon these summaries covering aspects of coherence, relevance, consistency and fluency.”
The paper relies on this without support.
The five meta-evaluation datasets are representative of NLG tasks and suitable for evaluating general NLG metrics.
“We conduct experiments on five NLG metaevaluation datasets (including summarization, story generation and data-to-text tasks).”
The paper relies on this without support.
For a given beginning of a story, various generated storylines and different plots could satisfy people.
“for a given beginning of a story, various generated storylines and different plots could satisfy people.”
The paper relies on this without support.
Human judgments are a valid gold standard for evaluating NLG metrics.
“ChatGPT achieves the new state-of-theart correlations (with human judgments) on Sum-mEval and OpenMEVA meta-evaluation datasets”
The paper relies on this without support.
The task-specific and aspect-specific prompts used are adequate to elicit ChatGPT's NLG evaluation capability.
“To this end, we design task-specific as well as aspectspecific prompts to guide ChatGPT to perform as an NLG metric.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“GPT-3.5 has been proved to have positive correlation with human annotations (Chiang and Lee, 2023a;Wang et al., 2023a)”
From Bayesian Calibration of Win Rate Estimation with LLM Evaluators (2024) · cites this paper as Wang et al., 2023a
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Recent studies have demonstrated that large language models (LLMs) can serve as reliable evaluators for natural language generation (NLG) tasks, exhibiting strong correlation with human judgments.”
From Enhancing memory retrieval in generative agents through LLM-trained cross attention networks (2025) · cites this paper as Wang et al. (2023)
In the citing paper
“Some other evaluators based on LLMs [37,60,74,118] also show good human alignment in more NLG tasks, especially compared with traditional metrics.”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [118]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.4 Story Generation
“ChatGPT (DA w/o ref) achieves 0.507 sample-level Spearman correlation, while PPL only achieves 0.324 sample-level Spearman correlation.”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
“| ChatGPT (DA w/o ref) ChatGPT (Stars w/o ref) 0.472 0.490 0.427 0.415 0.439 0.342 0.507 0.533 0.439 0.471 0.494 0.366 | |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| ROUGE-2 | 0.026 -0.108 0.009 0.091 -0.056 0.065 0.079 -0.087 0.052 0.048 -0.101 0.032 0.061 -0.088 0.092 |”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > Limitations
Reported 0.092 · recomputed 0.0395
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.1 Experimental Setup
“Two widely-used correlation measures are adopted:”
§ Is ChatGPT a Good NLG Evaluator? A Preliminary Study > 4.1 Experimental Setup
“(3) Kendall's Tau (Kendall, 1938) evaluates the ordinal association between two measured quantities.”
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
(2004) · Cited by 9,718
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
A NEW MEASURE OF RANK CORRELATION
Kendall, M. G. (1938) · Biometrika · doi:10.1093/biomet/30.1-2.81 · Cited by 7,299
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 2,459
A New Measure of Rank Correlation
Kendall, M. G. (1938) · Biometrika · doi:10.2307/2332226 · Cited by 2,346
From Word Embeddings To Document Distances
Kusner, M. J., Sun, Y., Kolkin, N., & Weinberger, K. Q. (2015) · PolyPublie (École Polytechnique de Montréal) · Cited by 1,695
Research works printed but not linked: 3
- Mavuto M Mukaka. 2012. A guide to appropriate use of correlation coefficient in medical research. Malawi medical journal, 24(3):69-71.
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: A method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computa- tional Linguistics, ACL '02, page 311-318, USA. Association for Computational Linguistics.
- Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023a. Cross-lingual summarization via chatgpt. arXiv preprint.
Cited by
Sorted by citations
Items 101–200 of 245
Kim, J., Koo, B., Nam, M., Jang, K., Lee, J., Chung, M., & Song, Y. B. (2025) · Applied Sciences · doi:10.3390/app15063355 · Cited by 9
Enhancing memory retrieval in generative agents through LLM-trained cross attention networks
Hong, C., & He, Q. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1591618 · Cited by 7
Detection of Hate Speech, Racism and Misogyny in Digital Social Networks: Colombian Case Study
Moreno-Sandoval, L. G., Pomares, A., Barbosa-Sierra, S. A., & Rojas, L. M. P. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8090113 · Cited by 12
C3AI: Crafting and Evaluating Constitutions for Constitutional AI
Kyrychenko, Y., Zhou, K., Bogucka, E. P., & Quercia, D. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714705 · Cited by 18
Generative Models For Indic Languages: Evaluating Content Generation Capabilities
Bhat, S., Varma, V., IIIT Hyderabad Gachibowli, H. 5., & Pedanekar, N. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_021 · Cited by 7
Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model
Jia, Q., Ren, S., Liu, Y., & Zhu, K. Q. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.679 · Cited by 20
GRACE: Discriminator-Guided Chain-of-Thought Reasoning
Khalifa, M., Logeswaran, L., Lee, M., Lee, H., & Wang, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1022 · Cited by 31
A Coarse-to-Fine Framework for Entity-Relation Joint Extraction
Zhang, M., Wang, J., Qu, J., Li, Z., Liu, A., Zhao, L., Chen, Z., & Zhou, X. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00082 · Cited by 7
Self-citationLeveraging Large Language Models for Improving Keyphrase Generation for Contextual Targeting
Bai, X., Wu, X. M., Stojković, I., & Tsioutsiouliklis, K. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3680093 · Cited by 9
GPT-4 as a Twitter Data Annotator: Unraveling Its Performance on a Stance Classification TaskPreprint
Liyanage, C., Gokani, R., & Mago, V. (2023) · doi:10.36227/techrxiv.24143706.v1 · Cited by 7
Self‐regulation, corruption, and competitiveness in extractive industries: Making transparency pay
Tang, S., Elfenbein, D. W., & Pasipanodya, T. M. G. (2025) · Strategic Management Journal · doi:10.1002/smj.70001 · Cited by 6
Zhang, W., Chen, Y., Liu, X., & Zhang, H. (2025) · Land Use Policy · doi:10.1016/j.landusepol.2025.107807 · Cited by 7
MCeT: Behavioral Model Correctness Evaluation using Large Language Models
Ahmed, K. A. M., Song, J., Chen, B., Ou, W., & Zheng, B. (2025) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1109/models67397.2025.00014 · Cited by 7
A Question-Answering Approach to Evaluating Legal Summaries
Xu, H., & Ashley, K. D. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230977 · Cited by 6
Liu, L., Qin, T., Zhou, Y., Wang, C., & Guan, X. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3397961 · Cited by 8
TwinStar: A Novel Design for Enhanced Test Question Generation Using Dual-LLM Engine
Zhuge, Q., Wang, H., & Chen, X. (2025) · Applied Sciences · doi:10.3390/app15063055 · Cited by 6
Kaswar, A. B., Nurjannah, N., Arsyad, M., Surianto, D. F., & Rosidah, R. R. (2023) · Vokatek Jurnal Pengabdian Masyarakat · doi:10.61255/vokatekjpm.v1i3.248 · Cited by 6
Improving Machine Translation Formality with Large Language Models
Yang, M., & Li, F. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.058248 · Cited by 9
Huang, Y., Fung, T., & DeLaurentis, D. A. (2025) · Systems Engineering · doi:10.1002/sys.70012 · Cited by 5
Liu, D., Long, Y., Zuoqiu, S., Liu, D., Li, K., Lin, Y., Liu, H., Yin, R., & Tang, T. (2025) · Journal of Medical Internet Research · doi:10.2196/85206 · Cited by 5
Can Artificial Intelligence help provide more sustainable feed-back?
Puertas, E., & Cano, E. (2024) · Digital Education Review · doi:10.1344/der.2024.45.50-58 · Cited by 5
MATEval: A Multi-agent Discussion Framework for Advancing Open-Ended Text Evaluation
Li, Y., Zhang, S., Wu, R. M., Huang, X., Chen, Y., Xu, W., Qi, G., & Min, D. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5575-2_31 · Cited by 12
Learning Evaluation Models From Large Language Models for Sequence Generation
Wang, C., Zhou, H., Chang, K., Liu, T., Zhang, C., Yang, M., Du, Q., Xiao, T., Zhang, Y., & Zhu, J. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3587460 · Cited by 11
Jin, W., Zhao, B., & Liu, G. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-8145-8_18 · Cited by 5
Laskar, M. T. R., Rahman, M., Jahan, I., Hoque, E., & Huang, J. X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.686 · Cited by 6
InterpreTutor: Using Large Language Models for Interpreter Assessment
Ünlü, C. (2023) · International Conference on Human-informed Translation and Interpreting Technology (HiT-IT) · doi:10.26615/issn.2683-0078.2023_007 · Cited by 8
How Do Hackathons Foster Creativity? Towards Automated Evaluation of Creativity at Scale
Falk, J., Chen, Y., Rafner, J. F., Zhang, M., Bjerva, J., & Nolte, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713447 · Cited by 6
Liang, J., Lin, Y., Liu, Y., Huang, Y., & Wu, H. (2025) · Building Simulation · doi:10.1007/s12273-025-1324-9 · Cited by 7
DeepTrans: Deep Reasoning Translation via Reinforcement Learning
Wang, J., Meng, F., & Zhou, J. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.65 · Cited by 4
Self-citationQiu, Z., Luo, L., Pan, S., & Liew, A. W. (2025) · Proceedings · doi:10.1109/icdm65498.2025.00159 · Cited by 10
LLMs are not for everything: A Dataset and Comparative Study on Argument Strength Classification
Djouvas, C., Charalampous, A., Christodoulou, C. J., & Tsapatsoulis, N. (2024) · Pan-Hellenic Conference on Progress in Computing and Informatics (PCI) · doi:10.1145/3716554.3716620 · Cited by 4
Dialogizer: Context-aware Conversational-QA Dataset Generation from Textual Sources
Hwang, Y., Kim, Y., Bae, H., Lee, H., Bang, J., & Jung, K. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.545 · Cited by 7
Understanding Faithfulness and Reasoning of Large Language Models on Plain Biomedical Summaries
Fang, B., Dai, X., & Karimi, S. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.578 · Cited by 4
Kunstmann, H., Ollier, J., Persson, J., & von Wangenheim, F. (2026) · ACM Transactions on Recommender Systems · doi:10.1145/3803546 · Cited by 3
Improving the Text Convolution Mechanism with Large Language Model for Review-Based Recommendation
Choi, Y., & Azad, F. T. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825014 · Cited by 3
Understanding the Effects of Explaining Predictive but Unintuitive Features in Human-XAI Interaction
Qu, J., Arguello, J., & Wang, Y. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732021 · Cited by 4
Self-citationAn Empirical Study of Evaluating Long-form Question Answering
Xian, N., Fan, Y., Zhang, R., de Rijke, M., & Guo, J. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729895 · Cited by 5
FORGE: An LLM-driven Framework for Large-Scale Smart Contract Vulnerability Dataset Construction
Chen, J., Shen, Y., Zhang, J., Li, Z., Grundy, J., SHAO, Z., Wang, Y., Wang, J., Chen, T., & Zheng, Z. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3764529 · Cited by 3
Self-citationA Multi-LLM Pipeline for Retrieval-Grounded, Bloom’s Taxonomy-Aligned Question Generation
Abdullah, M., Mansoor, I., Rey, V. F., & Fraz, M. M. (2025) · International Conference on Digital Futures and Transformative Technologies (ICoDT2) · doi:10.1109/icodt269104.2025.11360748 · Cited by 3
ScotRec: Social Chain-of-Thought LLM Reasoning for Recommendation
Li, K., Zou, J., Lin, Q., Guo, W., He, Q., & Yang, Y. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792313 · Cited by 4
Shi, H., & Sakai, T. (2024) · IEEE Access · doi:10.1109/access.2024.3378518 · Cited by 3
Self-citationReinforcement Learning with Large Language Models (LLMs) Interaction for Network Services
Du, H., Zhang, R., Niyato, D. T., Kang, J., Xiong, Z., & Kim, D. I. (2024) · 2012 International Conference on Computing, Networking and Communications (ICNC) · doi:10.1109/icnc59896.2024.10555960 · Cited by 6
QUDeval: The Evaluation of Questions Under Discussion Discourse Parsing
Wu, Y., Mangla, R., Durrett, G., & Li, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.325 · Cited by 7
Can Large Language Models Support Medical Facilitation Work? A Speculative Analysis
Abdulhamid, N. G., Ochieng, M., Bali, K., Ankrah, E. A., Karusala, N., Ronen, K., & O’Neill, J. (2023) · African Human Computer Interaction Conference (AfriCHI) · doi:10.1145/3628096.3628752 · Cited by 7
Exploring Prompting Large Language Models as Explainable Metrics
Mahmoudi, G. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.18 · Cited by 5
Optimizing the role of human evaluation in LLM-based spoken document summarization systems
Kroll, M., & Kraus, K. (2024) · Interspeech (USB) · doi:10.21437/interspeech.2024-2268 · Cited by 3
Can Large Language Models Assess Serendipity in Recommender Systems?
Tokutake, Y., Editorial Office, & Okamoto, K. (2024) · Journal of Advanced Computational Intelligence and Intelligent Informatics · doi:10.20965/jaciii.2024.p1263 · Cited by 3
GPT-4 as a Twitter Data Annotator: Unraveling Its Performance on a Stance Classification TaskPreprint
Liyanage, C., Gokani, R., & Mago, V. (2023) · doi:10.36227/techrxiv.24143706 · Cited by 3
Large Language Models Evaluate Machine Translation via Polishing
Wang, Y. (2023) · International Conference on Algorithms, Computing and Artificial Intelligence (ACAI) · doi:10.1145/3639631.3639658 · Cited by 2
Huang, C.-Y., Shih, M.-H., Lu, K.-H., Hsiao, C.-Y., & Lee, H.-Y. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10889205 · Cited by 2
DSciSum: Detailed summarization of long scientific documents
Liu, R., Mao, X.-L., & Huang, H. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113409 · Cited by 2
Saadany, H., Orǎsan, C., Breslin, C. J., Barczentewicz, M., & Walker, S. (2025) · Applied Sciences · doi:10.3390/app15169205 · Cited by 3
Yazma becerisini yapay zekâ odağında yeniden düşünmek: İmkân ve ihtimaller
Atasoy, A. (2025) · Alanyazın · doi:10.59320/alanyazin.1781293 · Cited by 2
Methods and evaluation in unsupervised keyphrase prediction: a survey
Han, Y., Wu, H., & Sun, Y. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-01001-9 · Cited by 2
Towards Efficient Coarse-grained Dialogue Response Selection
Lan, T., Mao, X.-L., Wei, W., Gao, X., & Huang, H. (2023) · ACM Transactions on Information Systems · doi:10.1145/3597609 · Cited by 2
Zhang, Z., Cheng, J., Sun, H., Deng, J., & Huang, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.700 · Cited by 2
Mögliche Einsatzgebiete von Künstlicher Intelligenz im Verlagswesen
Klotz, D. (2024) · Springer eBooks · doi:10.1007/978-3-658-43037-5_1 · Cited by 2
Sentiment and Interest Detection in Social Media using GPT-based Large Language Models
Asad, M. A. A., Imran, H. M., Alamin, M., Abdullah, T. A., & Chowdhury, S. I. (2023) · International Conference on Machine Learning and Natural Language Processing (MLNLP) · doi:10.1145/3639479.3639523 · Cited by 2
Persona-Identified Chatbot through Small-Scale Modeling and Data Transformation
Keum, B., Sun, J., Lee, W., Park, S., & Kim, H. (2024) · Electronics · doi:10.3390/electronics13081409 · Cited by 2
Kim, Y., Hwang, Y., Bae, H., Kang, T., & Jung, K. (2024) · IEEE Access · doi:10.1109/access.2024.3409377 · Cited by 3
Hegde, R. R., & Sharma, S. (2024) · International Conference on AI-ML Systems (AIMLSystems) · doi:10.1145/3703412.3703421 · Cited by 2
Giabbanelli, P. J., Daumas, C., Flandre, N. Y. J.-P., Pitkar, A., & Vazquez-Estrada, J. (2025) · Journal of Simulation · doi:10.1080/17477778.2025.2536663 · Cited by 2
Osipjan, A., Khorashadizadeh, H., Kessel, A.-L., Groppe, S., & Groppe, J. (2025) · Computers · doi:10.3390/computers14090382 · Cited by 4
ClozEx: A Task toward Generation of English Cloze Explanation
Zhang, Z., Mita, M., & Komachi, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.347 · Cited by 3
Well Begun is Half Done: Generator-agnostic Knowledge Pre-Selection for Knowledge-Grounded Dialogue
Qin, L., Zhang, Y., Liang, H., Wang, J., & Yang, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.285 · Cited by 3
Self-citationA Review on the Perks of Using ChatGPT in EducationPreprint
Heathen, T. S., & Lin, D. E. (2024) · Preprints.org · doi:10.20944/preprints202406.1060.v1 · Cited by 2
OWLed: Outlier-weighed Layerwise Pruning for Efficient Autonomous Driving Framework
Li, J., Yin, L., & Wang, X. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227233 · Cited by 2
Sánchez, A. M. (2025) · Perspectives · doi:10.1080/0907676x.2025.2600607 · Cited by 1
Mandapati, S. V., Kummetha, V. C., Concas, S., & Staes, L. (2025) · Electronics · doi:10.3390/electronics15010152 · Cited by 1
Community Needs and Assets: A Computational Analysis of Community Conversations
Chowdhury, M. T. A., Sharma, N. K., & KhudaBukhsh, A. R. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31315 · Cited by 2
The Accuracy of Domain Specific and Descriptive Analysis Generated by Large Language Models
Otieno, D. O., Abri, F., Siami‐Namini, S., & Namin, A. S. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00274 · Cited by 2
LLM-Based Empathetic Response Through Psychologist-Agent Debate
Wu, Y., Feng, S., Ming, W., Wang, D., & Zhang, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-7232-2_14 · Cited by 3
Huang, J., Wei, Y., Zhang, L., & Chen, W. (2024) · International Symposium on Educational Technology · doi:10.1109/iset61814.2024.00022 · Cited by 1
ArgueMapper Assistant: Interactive Argument Mining Using Generative Language Models
Lenz, M., & Bergmann, R. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77915-2_14 · Cited by 2
Meta-evaluating the Effects of Social Preferences on NPC-evaluators in an Energy Community Game
Isaza-Giraldo, A., Bala, P., Jiskrová, A., Sachser, L., Campos, P., & Pereira, L. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3720218 · Cited by 1
Zhao, K., Yang, B., Tang, C., Dai, S., Tang, H., Lin, C., & Zhan, L. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109574 · Cited by 2
Multi-Chain of Thought Prompt Learning for Aspect-Based Sentiment Analysis
He, Y., He, Z., Gu, T., Gu, B., Wan, Y., & Li, M. (2025) · Applied Sciences · doi:10.3390/app152212225 · Cited by 3
Autonomous Question and Answer System Based on ChatGPT With Large Language ModelPreprint
Wang, J. (2024) · Qeios · doi:10.32388/3n00oc · Cited by 1
Self-citationCompression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression
Johnson, W. (2026) · arXiv · Cited by 1
YESciEval: Robust LLM-as-a-Judge for Scientific Question Answering
D'Souza, J., Giglou, H. B., & Münch, Q. (2025) · arXiv · doi:10.48550/arxiv.2505.14279
Generating Faithful Text From a Knowledge Graph with Noisy Reference Text
Hashem, T., Wang, W., Wijaya, D. T., Ali, M. E., & Li, Y.-F. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.8
Stepping Stones for Self-Learning
Hai-Jew, S. (2023) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-0074-9.ch005
Bao, F. S., Tu, R., Luo, G., Yang, Y., Li, H., Qiu, M., He, Y., & Chen, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.87
Unveiling the Pros and Cons of Generative Ai Services: A Mixed-Methods ApproachPreprint
Yoon, S.-H., Yang, S., & Lee, S. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4770620
Tasks for LLMs and Their Evaluation
Ostapuk, N., & Audiffren, J. (2024) · Springer eBooks · doi:10.1007/978-3-031-54827-7_6 · Cited by 2
Guiding ChatGPT to Generate Salient Domain Summaries
Gao, J. H., Cao, Z., Huang, S., Qin, L., & Ai, C. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650616 · Cited by 2
Shen, Z., Li, W., & Shao, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9437-9_8
Inoshita, K. (2024) · IEEE International Conference on Internet of Things and Intelligence Systems (IoTaIS) · doi:10.1109/iotais64014.2024.10799461
Ma, M., Yang, Y., Yang, H., Deng, X., Fan, Q., & Feng, Y. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10821718
Development and evaluation of a semi-autonomous parallel attentive listening system
Lala, D., Inoue, K., Kawai, H., Pang, Z. H., Elmers, M., & Kawahara, T. (2024) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc63619.2025.10848649
Dong, W., Chen, S., Gu, M. G., & Yang, Y. T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88714-7_8
Agent-Based ChatGPT for Multi-Party Conversation Task Processing
万, 静. (2025) · Software Engineering and Applications · doi:10.12677/sea.2025.142022
Investigating Effects of Variation in Evaluating Dialogue Quality Using Large Language Models
Irum, A., & Beg, M. O. (2024) · International Multi-Topic Conference · doi:10.1109/inmic64792.2024.11004416
Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications
Tavakoli, L., & Zamani, H. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744574 · Cited by 4
Yang, D., Zhang, X., Han, T., & Liu, Y. (2025) · Natural language processing. · doi:10.1017/nlp.2025.10005 · Cited by 1
MELLM - An Automatic Evaluation Framework for LLM Without Human Supervision
Cui, C., Zhu, L., Ning, H., Sheng, X., & Wu, H. (2024) · International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE) · doi:10.1109/icbaie63306.2024.11116990
Chen, X., Lan, T., Lv, Z., Dong, Q., Zhang, J., Huang, H., Yang, M., & Hu, B. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.129878 · Cited by 1
Leveraging Large Language Models for Complementary Product Ads Recommendation
Jeon, B. E., Bae, R., & Bai, X. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3760940
Data Profile Generation Framework for Data Utilization
Matsumoto, I., Matsui, T., Fujita, Y., Suwa, H., & Yasumoto, K. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-4575-9_6
Think-to-Detect: Rationale-Driven Vision–Language Anomaly Detection
Abdalla, M. I., Kasem, M. S., Mahmoud, M., Senussi, M. F., Abdallah, A., & Kang, H. (2025) · Mathematics · doi:10.3390/math13243920
Metadata
- Authors
- Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J.
- Affiliations
- Soochow University (CN)
- Beijing Jiaotong University (CN)
- Tencent (China) (CN)
- Waseda University (JP)
- Fudan University (CN)
- Venue
- Proceedings of the 4th New Frontiers in Summarization Workshop
- Published
- 2023
- Pages
- 1-11
- OpenAlex
- W4389519239
- Cited by
- 235
- References
- 42
- Citations by year
- 2026: 43 · 2025: 65 · 2024: 75 · 2023: 52
- FWCI
- 41.64
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Workshop on New Frontiers in Summarization (NewSum) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- ChatGPT, natural language generation, automatic evaluation metrics, large language models, human judgment correlation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
