G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment
Open accessAbstract
The quality of texts generated by natural language generation (NLG) systems is hard to measure automatically.Conventional referencebased metrics, such as BLEU and ROUGE, have been shown to have relatively low correlation with human judgments, especially for tasks that require creativity and diversity.Recent studies suggest using large language models (LLMs) as reference-free metrics for NLG evaluation, which have the benefit of being applicable to new tasks that lack human references.However, these LLM-based evaluators still have lower human correspondence than medium-size neural evaluators.In this work, we present G-EVAL, a framework of using large language models with chain-of-thoughts (CoT) and a form-filling paradigm, to assess the quality of NLG outputs.We experiment with two generation tasks, text summarization and dialogue generation.We show that G-EVAL with GPT-4 as the backbone model achieves a Spearman correlation of 0.514 with human on summarization task, outperforming all previous methods by a large margin.We also propose analysis on the behavior of LLM-based evaluators, and highlight the potential concern of LLM-based evaluators having a bias towards the LLM-generated texts. 1
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
G-Eval ensured an objective comparison of the frameworks' ability to generate high-quality outputs.
“This scoring method ensured an objective comparison of the frameworks' ability to generate high-quality outputs.”
From AgentGit: A Version Control Framework for Reliable and Scalable LLM-Powered Multi-Agent Systems (2025) · cites this paper as Liu et al. 2023
Section 3.2 describes using G-EVAL to assess generated-text quality, but Section 4 Analysis contradicts the claim that it ensured an objective comparison: G-EVAL-4 favored GPT-3.5 summaries even when human judges preferred human-written ones. The Limitations section also warns of bias toward LLM-generated text. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredLLM-as-a-Judge models can approximate the depth and consistency of manual human evaluation while providing better consistency and stability.
“These models have shown they can approximate the depth and consistency of manual human evaluation, but also provide better consistency and stability.”
From LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints (2024) · cites this paper as Liu et al., 2023
The Introduction and Conclusion report that G-EVAL’s scores correlate well with human judgments on summarization and dialogue-generation benchmarks, supporting the claim that it can approximate some human evaluations. They do not establish that LLM-as-a-Judge models match the depth of manual evaluation or provide better consistency and stability than human evaluators. Section 4 also cautions that G-EVAL may favor LLM-generated text. (LLM, conf=0.85)
Experimental — accuracy not yet measuredGPT-4 is employed for automated evaluation of generated hypotheses and provided background across the four metrics.
“we employ GPT-4 for automated evaluation, where both the generated hypotheses and the provided background are evaluated across these aspects.”
From Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation (2024) · cites this paper as Liu et al., 2023
The Method and Implementation Details sections describe G-EVAL as an automated evaluator that uses GPT-4. The Benchmarks section describes four evaluation aspects for summaries and four for dialogue responses, but the paper does not evaluate generated hypotheses or provided background across the citing paper’s four metrics. No page numbers are available in our copy. (LLM, conf=0.84)
Experimental — accuracy not yet measuredA fixed LLM Judge can evaluate relevance, specificity, actionability, and overall quality of suggestions, and chain consistency.
“A fixed LLM Judge (Liu et al. 2023;Zheng et al. 2023) additionally evaluates the paper, predicted critique, and generated suggestion for relevance, specificity, actionability, and overall quality on a 1-5 scale, where higher scores are better.”
From Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review (2026) · cites this paper as Liu et al. 2023
The Method and Auto Chain-of-Thoughts for NLG Evaluation sections describe an LLM evaluator that scores generated text against specified criteria, including a 1–5 example. They do not establish evaluation of paper critiques or suggestions for specificity, actionability, overall quality, or chain consistency; the paper’s chain-of-thought steps are not a chain-consistency criterion. (LLM, conf=0.86)
Experimental — accuracy not yet measuredIn domains where no programmatic verifier exists, the community has converged on prompt-specific rubrics with weighted criteria, each scored by an LLM judge whose aggregate judgment serves as the scalar reward.
“In domains where no programmatic verifier exists (medical advice, scientific writing, instruction following), the community has converged on prompt-specific rubrics with weighted criteria, each scored by an LLM judge whose aggregate judgment serves as the scalar reward”
From Do You Need a Frontier Model as a Citation Verifier? Benchmarking Rubric LLMs for Deep-Research Source Attribution (2026) · cites this paper as Liu et al., 2023
The Method and Auto Chain-of-Thoughts sections (page not provided) describe an LLM judge using task-specific evaluation criteria to produce a score. But the paper weights possible scores by token probability, not rubric criteria. It does not establish community convergence across the named domains or report aggregate judgments being used as rewards; the Introduction discusses reward use only as a possibility. (LLM, conf=0.87)
Experimental — accuracy not yet measured
The LLM-as-Judge evaluation strategy is widely adopted and has been demonstrated to align well with human preferences.
“This LLM-as-Judge evaluation strategy is widely adopted in both NLP and software engineering domains [35], [36], as it has been demonstrated to align well with human preferences and provides more nuanced evaluation compared to traditional metrics [37], [38].”
From LongCodeZip: Compress Long Context for Code Language Models (2025) · cites this paper as [35]
The Introduction says using powerful LLMs to evaluate other models was becoming popular, and the Conclusion reports that G-EVAL achieved higher correspondence with human judgments than other evaluators. These passages support adoption in NLP and human alignment for G-EVAL, but do not establish adoption in software engineering or that LLM-as-Judge strategies generally align well with human preferences. The paper also cautions that LLM-based evaluators can be biased toward LLM-generated text. (LLM, conf=0.86)
Experimental — accuracy not yet measuredLLM-based evaluation can achieve a high correlation with human judgments and can be used to evaluate explanation quality, coherence, and overall usefulness of generated content.
“Studies have shown that LLM-based evaluation can achieve a high correlation with human judgments and can be used to evaluate explanation quality, coherence, and overall usefulness of generated content [34].”
From An Integrated AHP–Fuzzy AHP Evaluation Framework for Large Language Models in Software Engineering Education (2026) · cites this paper as [34]
The Introduction (page unavailable) reports that G-EVAL’s scores correlate strongly with human quality judgments, and the Evaluation Criteria section explicitly includes coherence. The paper evaluates summaries and dialogue responses, but the passages do not establish that it evaluates explanation quality or overall usefulness as distinct criteria. (LLM, conf=0.88)
Experimental — accuracy not yet measuredLLMs can provide scalable and rubric-aware evaluations for summarization, dialogue, and long-form responses, often showing stronger correlations with human judgments than traditional metrics.
“Prior studies have shown that LLMs can provide scalable and rubric-aware evaluations for summarization, dialogue, and long-form responses, often showing stronger correlations with human judgments than traditional metrics (Liu et al., 2023; Zhu et al., 2025; Kim et al., 2024).”
From Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation (2026) · cites this paper as Liu et al., 2023
The Introduction and Conclusion describe G-EVAL as an evaluation method guided by specified criteria and report stronger correlations with human judgments for summarization and dialogue generation than comparison metrics. The paper does not report evaluations of long-form responses or establish that the method is scalable; those parts of the citing sentence are not supported by this source. (LLM, conf=0.87)
Experimental — accuracy not yet measuredGPT-4 with chain-of-thought prompts aligns more closely with human judgments than conventional automatic metrics on NLG tasks.
“Notably, GPT-4 with chainof-thought prompts aligns more closely with human judgments than conventional automatic metrics on NLG tasks (Liu et al., 2023; OpenAI et al., 2024; Wei et al., 2022).”
From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Liu et al., 2023
Liu et al. (2023) took advantage of variability in GPT-4 by setting temperature to 1 and sampling 20 times to get multiple scores and then getting the final rating by normalizing all the scores by their probabilities.
“Liu et al. (2023) took advantage of this variability in GPT-4 by setting temperature to 1 and sampling 20 times to get multiple scores and then getting the final rating by normalizing all the scores by their probabilities.”
From Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks (2025) · cites this paper as Liu et al., 2023
Traditional NLP metrics such as ROUGE, METEOR, and BLEU are known to poorly correlate with human evaluations of natural language generations.
“Traditional NLP metrics such as ROUGE, METEOR, and BLEU—which simply count n-gram overlap between generated and reference text—are known to poorly correlate with human evaluations of natural language generations83–85.”
From The shaky foundations of large language models and foundation models for electronic health records (2023) · cites this paper as [85]
LLM judgments can align substantially with human preferences on open-ended generation tasks.
“Rubric-conditioned scoring in G-Eval and pairwise comparison in MT-Bench demonstrated that LLM judgments can align substantially with human preferences on open-ended generation tasks(Liu et al., 2023;Zheng et al., 2023).”
From How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review (2026) · cites this paper as Liu et al., 2023
LLM ratings of text quality track human ratings.
“a growing literature finds that LLM ratings of text quality track human ratings (Chiang and Lee, 2023; Liu et al., 2023; Zheng et al., 2023)”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Liu et al., 2023
The remarkable instruction-following ability of LLMs has underscored their potential for automatic evaluation.
“Recently, the remarkable instruction-following ability of LLMs has underscored their potential for automatic evaluation [51, 190, 314].”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [190]
Several rank correlations are reported together for graded or continuous scores.
“several rank correlations are reported together for graded or continuous scores (e.g., Liu et al., 2023;Ye et al., 2024;Kim et al., 2024)”
From Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why (2026) · cites this paper as Liu et al., 2023
Frameworks such as G-Eval obtain strong correlations with human ratings on summarization and dialog through structured rubrics and chain of thought prompting.
“while frameworks such as G-Eval (Liu et al., 2023) obtain strong correlations with human ratings on summarization and dialog through structured rubrics and chain of thought prompting.”
From Judge Reliability Harness: Stress Testing the Reliability of LLM Judges (2026) · cites this paper as Liu et al., 2023
The dimensions of clarity and insightfulness are commonly used in long-form generation evaluation.
“These dimensions are commonly used in long-form generation evaluation (Liu et al., 2023; Saha et al., 2024) and provide evidence of the presentation quality of the generated content.”
From DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research (2025) · cites this paper as Liu et al., 2023
The Evaluation Criteria section discusses coherence and engagingness, but the passages found do not identify clarity and insightfulness as evaluation dimensions. Searches for both terms and for their use in long-form generation evaluation found no passage establishing the attributed claim. (LLM, conf=0.70)
Experimental — accuracy not yet measuredOn summarization tasks, GPT-4 as evaluator achieves higher correlation with humans than other methods by a large margin.
“On summarization tasks, GPT-4 as an evaluator achieves a higher correlation with humans than other methods with a large margin [74].”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [74]
LLM evaluators may have a bias towards LLM-generated texts.
“But the LLM may have a bias towards the LLM-generated texts [74].”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [74]
Some LLM-based evaluators show good human alignment in more NLG tasks, especially compared with traditional metrics.
“Some other evaluators based on LLMs [37,60,74,118] also show good human alignment in more NLG tasks, especially compared with traditional metrics.”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [74]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
(2004) · Cited by 9,718 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Self-citationChain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
(2005) · Cited by 4,152 · see citing works
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,779 · see citing works
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
From Word Embeddings To Document Distances
Kusner, M. J., Sun, Y., Kolkin, N., & Weinberger, K. Q. (2015) · PolyPublie (École Polytechnique de Montréal) · Cited by 1,695 · see citing works
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207 · see citing works
Evaluating the Factual Consistency of Abstractive Text Summarization
Kryscinski, W., McCann, B., Xiong, C., & Socher, R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.750 · Cited by 669 · see citing works
Is ChatGPT a Good NLG Evaluator? A Preliminary StudyPreprint
Wang, J., Liang, Y.-L., Meng, F.-D., Sun, Z., Shi, H., Li, Z.-X., Xu, J.-A., Qu, J.-F., & Zhou, J. (2023) · NEWSUM · doi:10.48550/arxiv.2303.04048 · Cited by 669 · see citing works
SummEval: Re-evaluating Summarization Evaluation
Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00373 · Cited by 587 · see citing works
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C. M., & Eger, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1053 · Cited by 490 · see citing works
GPTScore: Evaluate as You Desire
Fu, J., Ng, S.-K., Jiang, Z., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2302.04166 · Cited by 446 · see citing works
Benchmarking Large Language Models for News Summarization
Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. B. (2023) · arXiv · doi:10.48550/arxiv.2301.13848 · Cited by 415 · see citing works
Benchmarking Large Language Models for News Summarization
Zhang, T., Ladhak, F., Durmus, E., Liang, P., McKeown, K., & Hashimoto, T. B. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00632 · Cited by 415 · see citing works
Faithful to the Original: Fact Aware Neural Abstractive Summarization
Cao, Z., Wei, F., Li, W., & Li, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11912 · Cited by 359 · see citing works
(2020) · doi:10.18653/v1/2020.acl-main.450 · Cited by 356 · see citing works
BARTScore: Evaluating Generated Text as Text Generation
Yuan, W., Neubig, G., & Liu, P. (2021) · arXiv · doi:10.48550/arxiv.2106.11520 · Cited by 353 · see citing works
Durmus, E., He, H., & Diab, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.454 · Cited by 306 · see citing works
Is chatGPT a good nlg evaluator? A preliminary study
Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J. (2023) · Proceedings of the 4th New Frontiers in Summarization Workshop · doi:10.18653/v1/2023.newsum-1.1 · Cited by 235 · see citing works
Reiter, E., & Belz, A. (2009) · Computational Linguistics · doi:10.1162/coli.2009.35.4.35405 · Cited by 204 · see citing works
Lu, Q., Qiu, B., Ding, L., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2303.13809 · Cited by 164 · see citing works
USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation
Mehri, S., & Eskenazi, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.64 · Cited by 149 · see citing works
Sentence Mover’s Similarity: Automatic Evaluation for Multi-Sentence Texts
Clark, E., Çelikyılmaz, A., & Smith, N. A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1264 · Cited by 138 · see citing works
Factual Error Correction for Abstractive Summarization Models
Cao, M., Dong, Y., Wu, J., & Cheung, J. C. K. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.506 · Cited by 132 · see citing works
Evaluating Evaluation Methods for Generation in the Presence of Variation
Stent, A., Marge, M., & Singhai, M. (2005) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-30586-6_38 · Cited by 126 · see citing works
Large Language Models Are State-of-the-Art Evaluators of Translation Quality
Kocmi, T., & Federmann, C. (2023) · arXiv · doi:10.48550/arxiv.2302.14520 · Cited by 122 · see citing works
Towards a Unified Multi-Dimensional Evaluator for Text Generation
Zhong, M., Liu, Y., Yin, D., Mao, Y., Jiao, Y., Liu, P., Zhu, C., Ji, H., & Han, J. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.131 · Cited by 117 · see citing works
Self-citationEvaluating Coherence in Dialogue Systems using Entailment
Dziri, N., Kamalloo, E., Mathewson, K. W., & Zai͏̈ane, O. R. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1381 · Cited by 91 · see citing works
Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings
Ghazarian, S., Wei, J. T.-Z., Galstyan, A., & Peng, N. (2019) · doi:10.18653/v1/w19-2310 · Cited by 84 · see citing works
Lu, Q., Qiu, B., Ding, L., Xie, L., & Tao, D. (2023) · Preprints.org · doi:10.20944/preprints202303.0255.v1 · Cited by 71 · see citing works
Large Language Models are not Fair Evaluators
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., & Sui, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.17926 · Cited by 34 · see citing works
Towards Quantifiable Dialogue Coherence Evaluation
Zheng, Y., Lu, L., Huang, L., Lin, L., & Liang, X. (2021) · doi:10.18653/v1/2021.acl-long.211 · Cited by 20 · see citing works
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand
Kasai, J., Sakaguchi, K., Bras, R. J. L., Dunagan, L., Morrison, J., Fabbri, A. R., Choi, Y., & Smith, N. A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.259 · Cited by 19 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
Research works printed but not linked: 1
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguistics, pages 311-318.
Cited by
Sorted by citations
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
Augmenting large language models with chemistry tools
M. Bran, A., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00832-8 · Cited by 794 · see citing works
Harnessing the power of llms in practice: a survey on chatGPT and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649506 · Cited by 644 · see citing works
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.741 · Cited by 407 · see citing works
The shaky foundations of large language models and foundation models for electronic health records
Wornow, M., Xu, Y., Thapa, R., Patel, B., Steinberg, E., Fleming, S., Pfeffer, M. A., Fries, J., & Shah, N. H. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00879-8 · Cited by 324 · see citing works
LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., & Gao, J. (2023) · arXiv · doi:10.48550/arxiv.2306.00890 · Cited by 590 · see citing works
“Recent studies [25, 29, 52] raises the self-enhancement bias issue of LLM evaluation, i.e., favoring its own generations.”
BackgroundFoundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213 · see citing works
Factuality challenges in the era of large language models and opportunities for fact-checking
Augenstein, I., Baldwin, T., Cha, M., Chakraborty, T., Ciampaglia, G. L., Corney, D., DiResta, R., Ferrara, E., Hale, S., Halevy, A., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00881-z · Cited by 187 · see citing works
Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.11511 · Cited by 101 · see citing works
“A state-of-the-art LLM like GPT-4 (OpenAI, 2023) can be effectively used to generate such feedback (Liu et al., 2023b).”
BackgroundSelf-citationKamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00713 · Cited by 82 · see citing works
AlignScore: Evaluating factual consistency with a unified alignment function
Zha, Y., Yang, Y., Li, R., & Hu, Z. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.634 · Cited by 81 · see citing works
Benchmarking Cognitive Biases in Large Language Models as Evaluators
Koo, R., Lee, M., Raheja, V., Park, J. I., Kim, Z. M., & Kang, D. (2023) · arXiv · doi:10.48550/arxiv.2309.17012 · Cited by 72 · see citing works
Benchmarking Cognitive Biases in Large Language Models as Evaluators
Koo, R., Lee, M., Raheja, V., Park, J. I., Kim, Z. M., & Kang, D. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.29 · Cited by 70 · see citing works
LLM4Decompile: Decompiling Binary Code with Large Language Models
Tan, H., Luo, Q., Li, J., & Zhang, Y. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.203 · Cited by 67 · see citing works
Xu, P., Chen, X., Zhao, Z., & Shi, D. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-325054 · Cited by 63 · see citing works
SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization
Kim, H., Hessel, J., Jiang, L., West, P., Lu, X., Yu, Y., Zhou, P., Bras, R., Alikhani, M., Kim, G., et al. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.799 · Cited by 63 · see citing works
Exploring the Impact of Temperature on Large Language Models: Hot or Cold?
Li, L., Sleem, L., Gentile, N., Nichil, G., & State, R. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.07.135 · Cited by 62 · see citing works
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 57 · see citing works
“[82] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.”
BackgroundAssessing and understanding creativity in large language models
Zhao, Y., Zhang, R., Li, W., & Li, L. (2025) · Machine Intelligence Research · doi:10.1007/s11633-025-1546-4 · Cited by 57 · see citing works
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
Chan, C.-M., Chen, W., Su, Y., Yu, J., Xue, W., Zhang, S., Fu, J., & Liu, Z. (2023) · arXiv · doi:10.48550/arxiv.2308.07201 · Cited by 55 · see citing works
“The second part and the third part of Table 1 show the results of FairEval’s method and the results of our proposed method respectively.”
BackgroundMethodResultInfluentialXing, W., Song, Y., Li, C., Liu, Z., Zhu, W., & Oh, H. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13586 · Cited by 50 · see citing works
Hashemi, H., Eisner, J., Rosset, C., Van Durme, B., & Kedzie, C. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.745 · Cited by 47 · see citing works
Zambrano Chaves, J. M., Huang, S.-C., Xu, Y., Xu, H., Usuyama, N., Zhang, S., Wang, F., Xie, Y., Khademi, M., Yang, Z., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-58344-x · Cited by 45 · see citing works
“The data used to train our model in the second and third stages is a combination of rule-based and GPT-4 structured data, as summarized in Supplementary Table 7.”
Self-citationBaek, J., Jauhar, S. K., Cucerzan, S., & Hwang, S. J. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.342 · Cited by 46 · see citing works
Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions
Chen, H., Fang, Z., Singla, Y., & Dredze, M. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.182 · Cited by 31 · see citing works
A Field Guide to Automatic Evaluation of LLM-Generated Summaries
van Schaik, T. A., & Pugh, B. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3661346 · Cited by 30 · see citing works
TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models
Gekhman, Z., Herzig, J., Aharoni, R., Elkind, C., & Szpektor, I. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.127 · Cited by 33 · see citing works
Mindstorms in Natural Language-Based Societies of Mind
Zhuge, M., Liu, H., Faccio, F., Ashley, D. R., Csordás, R., Gopalakrishnan, A., Hamdi, A., Hammoud, H. A. A. K., Herrmann, V., Irie, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.17066 · Cited by 24
Mindstorms in Natural Language-Based Societies of Mind
Zhuge, M., Liu, H., Faccio, F., Ashley, D. R., Csordás, R., Gopalakrishnan, A., Hamdi, A., Hammoud, H. A. A. K., Herrmann, V., Irie, K., et al. (2025) · Computational Visual Media · doi:10.26599/cvm.2025.9450460 · Cited by 24 · see citing works
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models
Chuang, Y.-S., Xie, Y., Luo, H., Kim, Y., Glass, J., & He, P. (2023) · arXiv · doi:10.48550/arxiv.2309.03883 · Cited by 29 · see citing works
“We conduct an additional study of the quality of generated text using GPT4, given the fact that several prior studies Chiang & Lee (2023a); Liu et al. (2023) have shown the great potential of GPT-4 to serve as an alternative to human evaluation.”
MethodTowards lifelong dialogue agents via timeline-based memory management
Ong, K. T.-I., Kim, N., Gwak, M., Chae, H., Kwon, T., Jo, Y., Hwang, S.-W., Lee, D., & Yeo, J. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.435 · Cited by 18 · see citing works
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 17 · see citing works
Understanding the Effects of RLHF on LLM Generalisation and Diversity
Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., & Raileanu, R. (2023) · arXiv · doi:10.48550/arxiv.2310.06452 · Cited by 17 · see citing works
“…tasks include translation, question-answering, and other standard NLP tasks, as well as newer tasks such as using LLMs to simulate human annotators (Dubois et al., 2023; Mao et al., 2023; Liu et al., 2023b) or as content generators for improving other models (Peng et al., 2023; Wang et al., 2023).”
MethodFactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence
Joseph, S., Chen, L., Trienes, J., Göke, H., Coers, M., Xu, W., Wallace, B., & Li, J. J. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.459 · Cited by 17 · see citing works
UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor
Upadhyay, S., Pradeep, R., Thakur, N., Craswell, N., & Lin, J. (2024) · arXiv · doi:10.48550/arxiv.2406.06519 · Cited by 16 · see citing works
“The emergence of large language models (LLMs) such as GPT-4 [17] and Gemini [20] has presented a distinctive opportunity to automate the intricate task of data annotation [1, 3, 12, 13, 15, 19, 22, 23].”
MethodQGEval: Benchmarking Multi-dimensional Evaluation for Question Generation
Fu, W., Wei, B., Hu, J., Cai, Z., & Liu, J. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.658 · Cited by 16 · see citing works
“5-turbo and GPT-4-1106-preview as evaluators and implement evaluations using both direct prompts and G-EVAL (Liu et al., 2023), an evaluation method employing Chain-of-Thought (COT).”
MethodFActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W.-T., Koh, P. W., Iyyer, M., Zettlemoyer, L., & Hajishirzi, H. (2023) · arXiv · doi:10.48550/arxiv.2305.14251 · Cited by 14 · see citing works
“A series of concurrent work verifies the precision of the citations (attributions) provided by the model (Gao et al., 2022; Liu et al., 2023a; Yue et al., 2023; Gao et al., 2023).”
BackgroundMethodInfluentialTesting the use of a large language model (LLM) for performing data quality assessment
MacMaster, S., & Sinistore, J. (2025) · The International Journal of Life Cycle Assessment · doi:10.1007/s11367-024-02405-8 · Cited by 14 · see citing works
Ozmen, B. B., Singh, N., Shah, K., Berber, I., Singh, D., Pinsky, E., Rampazzo, A., & Schwarz, G. S. (2025) · Journal of Hand and Microsurgery · doi:10.1016/j.jham.2025.100293 · Cited by 11 · see citing works
Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks
Haldar, R., & Hockenmaier, J. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.1361 · Cited by 11 · see citing works
“Liu et al. (2023) took advantage of this variability in GPT-4 by setting temperature to 1 and sampling 20 times to get multiple scores and then getting the final rating by normalizing all the scores by their probabilities.”
InfluentialFaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs
Bao, F. S., Li, M., Qu, R., Luo, G., Wan, E., Tang, Y., Fan, W., Tamber, M. S., Kazi, S., Sourabh, V., et al. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers) · doi:10.18653/v1/2025.naacl-short.38 · Cited by 8 · see citing works
A Logical Fallacy-Informed Framework for Argument Generation
Mouchel, L., Paul, D., Cui, S., West, R., Bosselut, A., & Faltings, B. (2025) · Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2025.naacl-long.374 · Cited by 7 · see citing works
Finding Blind Spots in Evaluator LLMs with Interpretable Checklists
Doddapaneni, S., Khan, M. S. U. R., Verma, S., & Khapra, M. M. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.911 · Cited by 9 · see citing works
Lessons from the Trenches on Reproducible Evaluation of Language Models
Biderman, S., Schoelkopf, H., Sutawika, L., Gao, L., Tow, J., Abbasi, B., Aji, A. F., Ammanamanchi, P. S., Black, S., Clive, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.14782 · Cited by 9 · see citing works
“As we discuss in Section 4, these dynamics are severe enough that the Benchmark Lottery has become a real phenomenon, with benchmark selection itself becoming a target for optimization rather than a means of measuring genuine progress.”
InfluentialLarge Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation
Qi, B., Zhang, K., Tian, K., Li, H., Chen, Z.-R., Zeng, S., Hua, E., Jinfang, H., & Zhou, B. (2024) · arXiv · doi:10.48550/arxiv.2407.08940 · Cited by 5 · see citing works
LongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00020 · Cited by 7 · see citing works
“This LLM-as-Judge evaluation strategy is widely adopted in both NLP and software engineering domains [35], [36], as it has been demonstrated to align well with human preferences and provides more nuanced evaluation compared to traditional metrics [37], [38].”
Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Models
Lyu, C., Wu, M., & Aji, A. F. (2024) · arXiv · doi:10.48550/arxiv.2402.13887 · Cited by 4 · see citing works
Understanding Faithfulness and Reasoning of Large Language Models on Plain Biomedical Summaries
Fang, B., Dai, X., & Karimi, S. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.578 · Cited by 4 · see citing works
LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory
Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., & Yu, D. (2024) · arXiv · doi:10.48550/arxiv.2410.10813 · Cited by 3 · see citing works
“5 Mini Instruct 4B 0.652 0.324 50.3% ↓ (b) Long-context LLMs exhibit large QA performance drops on L ONG M EM E VAL S (column “S”), compared to the accuracy of answering the questions based on only the evidence sessions (column “Oracle”). employs a LLM to assess response quality (Liu et al., 2023).”
InfluentialPalmeira Ferraz, T., Mehta, K., Lin, Y.-H., Chang, H.-S., Oraby, S., Liu, S., Subramanian, V., Chung, T., Bansal, M., & Peng, N. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.458 · Cited by 3 · see citing works
“For evaluation, this strategy not only provides detailed insights into model performance across different aspects but also makes evaluations more objective and less ambiguous, as models may excel in some areas while underperforming in others.”
Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator
Liao, Y., Meng, Y., Wang, Y., Liu, H., Wang, Y., & Wang, Y. (2024) · arXiv · doi:10.48550/arxiv.2403.08495 · Cited by 15 · see citing works
SWE-Pruner: Self-Adaptive Context Pruning for Coding Agents
Wang, Y., Shi, Y., Yang, M., Zhang, R., He, S., Lian, H., Chen, Y., Ye, S., Cai, K., & Gu, X. (2026) · arXiv · doi:10.48550/arxiv.2601.16746 · Cited by 2 · see citing works
“Following initial generation, we apply a rigorous quality control mechanism using Qwen3-Next-80B-A3B-Thinking as an LLM-as-a-Judge filter (Wang et al., 2025a; Liu et al., 2023b; Song et al., 2024; He et al., 2025).”
LongCodeZip: Compress Long Context for Code Language Models
Shi, Y., Qian, Y., Zhang, H., Shen, B., & Gu, X. (2025) · arXiv · Cited by 7 · see citing works
“This LLM-as-Judge evaluation strategy is widely adopted in both NLP and software engineering domains [35], [36], as it has been demonstrated to align well with human preferences and provides more nuanced evaluation compared to traditional metrics [37], [38].”
Bayesian Calibration of Win Rate Estimation with LLM Evaluators
Gao, Y., Xu, G., Wang, Z., & Cohan, A. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.273 · Cited by 1 · see citing works
“Some works applied various prompting techniques to improve the accuracy of LLM evaluation, including chain of thought (Liu et al., 2023a), evaluation with explanation (Chiang and Lee, 2023b), multi-LLM discussion (Chan et al., 2023; Li et al., 2023), and calibration with human expert (Liu et al.,…”
MethodNo Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
Krumdick, M., Lovering, C., Reddy, V., Ebner, S., & Tanner, C. (2025) · arXiv · doi:10.48550/arxiv.2503.05061 · Cited by 1 · see citing works
“…(Zheng et al., 2023), explicit length de-biasing (Dubois et al., 2024), using an ensemble of judge models as a jury (Verga et al., 2024), creating rubrics (Liu et al., 2023), performing prompt optimization (Zhou et al., 2024), and using a pairwise-preference search method (Liu et al., 2024).”
DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research
Coelho, J., Ning, J., He, J., Mao, K., Paladugu, A., Setlur, P., Jin, J., Callan, J., Magalhães, J., Martins, B., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.19253 · Cited by 1 · see citing works
“In the next section, we describe our evaluation methodology for extending the use of Researchy Questions to benchmark deep research generation.”
Jović, J. L., Domazet, D. S., Vesić, N. O., Ranđelović, B. M., & Simjanović, D. J. (2026) · Mathematics · doi:10.3390/math14101637 · Cited by 1 · see citing works
A Scoping Review of LLM-as-a-Judge in Healthcare and the MedJUDGE Framework
Li, C., Akhtar, Z., Kwak, M., Ji, Y., Zhang, H., Obi, T., Ren, Y., Wu, X., Sivarajkumar, S., Lehmann, H. P., et al. (2026) · arXiv · doi:10.48550/arxiv.2604.25933 · Cited by 1 · see citing works
The Good, the Bad and the Constructive: Automatically Measuring Peer Review’s Utility for Authors
Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.1476 · Cited by 1 · see citing works
RealMem: Benchmarking LLMs in Real-World Memory-Driven Interaction
Bian, H., Yao, Z., Hu, S., Xu, Z., Zhang, S., Guo, Y., Yang, Z., Han, X., Wang, H., & Chen, R. (2026) · arXiv · doi:10.48550/arxiv.2601.06966
“To comprehensively assess retrieval quality, we adopt a hybrid evaluation strategy that combines standard quantitative metrics with LLM-based semantic judgments (Liu et al., 2023).”
AgentGit: A Version Control Framework for Reliable and Scalable LLM-Powered Multi-Agent Systems
Li, Y., Ping, S., Chen, X., Qi, X., Wang, Z., Luo, Y., & Zhang, X. (2025) · arXiv · doi:10.48550/arxiv.2511.00628
“To evaluate the quality of the final outputs generated by each framework, we employed G-Eval (Liu et al. 2023), a widely used evaluation metric for assessing the coherence, relevance, and overall quality of generated text.”
Self-GC: Self-Governing Context for Long-Horizon LLM Agents
Hao, X., Meng, H., Yin, X., Zhu, J., & Cao, C. (2026) · arXiv · doi:10.48550/arxiv.2607.00692
“…et al. 2023a; Zhou et al. 2024; Guo et al. 2026), and LLM-as-a-judge protocols such as G-Eval and JudgeLM support auditable rubric-based evaluation (Liu et al. 2023b; Zhu, Wang, and Wang 2023); we adopt such a judge to isolate a question these benchmarks do not, namely whether the retained context…”
Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judge
Spiliopoulou, E., Fogliato, R., Burnsky, H., Soliman, T., Ma, J., Horwood, G., & Ballesteros, M. (2025) · arXiv · doi:10.48550/arxiv.2508.06709
“HELM-Instruct itself aggregates prompts from diverse sources [Bai et al., 2022a, Perez et al., 2022, Geng et al., 2023, Team, 2023, Köpf et al., 2023, Wang et al., 2023, Gridfiti, 2023].”
GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations
Yang, J., Lai, K.-H., Wang, X., Chang, S., Harari, Y., & Gabrilovich, E. (2026) · arXiv · doi:10.48550/arxiv.2605.14498
“We adapt G-Eval [31] to the group-chat setting and assess synthesized dialogues along six dimensions chosen to reflect properties specific to multi-party interaction: Naturalness , Coher-ence , Diversity , Contextual Relevance , Momentum , and Engagingness , each scored from 0 to 5 by GPT-5 as…”
Continual Learning, Not Training: Online Adaptation For Agents
Jaglan, A., & Barnes, J. (2025) · arXiv · doi:10.48550/arxiv.2511.01093
“This mechanism mirrors interactive teaching, where support is adjusted based on the learner’s immediate needs (Liu et al., 2023; Jung et al., 2025).”
Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review
Yuan, S., Zhang, X., Ni, W., Xiao, T., Di, S., Zhu, J., Jiang, Z., Tan, R., Chen, L., & Ren, K. (2026) · arXiv · doi:10.48550/arxiv.2609.05947
“A fixed LLM Judge (Liu et al. 2023; Zheng et al. 2023) additionally evaluates the paper, predicted critique, and generated suggestion for relevance, specificity, actionability, and overall quality on a 1–5 scale, where higher scores are better.”
InfluentialQuantifying and Mitigating Self-Preference Bias of LLM Judges
Yang, J., Hu, Z., Qiu, C., Deng, Z., Jiao, X., & Zhou, T. (2026) · arXiv · doi:10.48550/arxiv.2604.22891
“Recently, LLM-as-a-Judge has emerged as a dominant paradigm for re-placing costly human evaluation[1, 2, 3, 4], playing a critical role in model alignment[5] and leaderboard construction [6].”
InfluentialPRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers
Loc, N. P. P., La Viet, T. H., Khanh, T. T., Nguyen, D. A., Pham, T. A. N., Nguyen, T., Chawla, N. V., Buntine, W., Wong, K.-S., Doan, K. D., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.26730
Leung, E., Lumer, E., Feld, C., Huber, A., Subbiah, V. K., & Paul, K. (2026) · arXiv · doi:10.48550/arxiv.2607.08700
“…verifier exists (medical advice, scientific writing, instruction following), the community has converged on prompt-specific rubrics with weighted criteria, each scored by an LLM judge whose aggregate judgment serves as the scalar reward [Rezaei et al., 2026, Mahmoud et al., 2026, Liu et al., 2023].”
InfluentialDeepResearchGym: A Free, Transparent, and Reproducible Sandbox for Deep Research
Coelho, J., Ning, J., He, J., Mao, K., Paladugu, A. S., Setlur, P., Jin, J., Callan, J., Magalhães, J., Martins, B., et al. (2026) · Proceedings of the 2026 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) · doi:10.1145/3805713.3820409
Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation
Garg, K., Shaik, F., Bandyopadhyay, S., & Caragea, C. (2025) · arXiv · doi:10.48550/arxiv.2508.14273
“By combining automated metrics with LLM-as-a-judge evaluations (Liu et al., 2023), we seek to provide a holistic understanding of the strengths and limitations of current state-of-the-art models.”
How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review
Li, M., Wang, C., Li, X., Zeng, X., Li, D., Shi, P., Zhou, D., & Zhou, T. (2026) · arXiv · doi:10.48550/arxiv.2608.08975
Experimental Evidence on the Learning Impact of Generative AI
Contractor, Z., & Reyes, G. (2026) · arXiv · doi:10.48550/arxiv.2607.08849
He, Y., Smirnov, I., Fedrushkov, D., Kovalchuk, S., & Revin, I. (2026) · arXiv · doi:10.48550/arxiv.2604.03742
“[10].”
The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors
Sadallah, A., Baumgärtner, T., Gurevych, I., & Briscoe, T. (2025) · arXiv · doi:10.48550/arxiv.2509.04484
“Alternatively, closed-source models such as GPT-4 (Liu et al., 2023) or specialized LLM-as-a-judge models (Kim et al., 2024) can rate generated reviews.”
Attributable by Construction: Claim-Anchored Provenance for Multi-Document Summarization
Guan, S. (2026) · arXiv · doi:10.48550/arxiv.2606.23989
“Quality is reported with ROUGE (Lin, 2004), BERTScore (Zhang et al., 2020), and an LLM-as-judge coverage score (Liu et al., 2023a; Zheng et al., 2023) for the frontier of §5.8; faithfulness with AlignScore (Zha et al., 2023), SummaC (Laban et al., 2022), and FActScore (Min et al., 2023).”
InfluentialMAPLE: Multi-Aspect Panels of LLM Evaluators for Open-Ended Questions
Jinji, M., Atarashi, K., Takeuchi, K., & Kashima, H. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.1351
“Given this limitation, recent work has explored the use of LLMs as scalable evaluators to score open-ended outputs or generate preference labels (Ding et al., 2023; Zheng et al., 2023; Chen et al., 2024; Liu et al., 2023).”
Akarajaradwong, P., Lertprasertphakorn, W., Chaksangchaichot, C., & Nutanong, S. (2026) · arXiv · doi:10.48550/arxiv.2605.25652
“…with n-gram overlap [Papineni et al., 2002, Lin, 2004], LLM-as-a-judge protocols that grade win rate or per-facet quality [Zheng et al., 2023, Liu et al., 2023, Gu et al., 2025], or task-tailored rubrics [Es et al., 2024, Laban et al., 2024, Akarajaradwong et al., 2025, Magesh et al., 2025].”
JudgeMoE: Distributional Aggregation for LLM-as-a-Judge
Liu, Y., James, J., Wang, Y., Zhao, K., Xiao, C., & Lin, C. (2026) · arXiv · doi:10.48550/arxiv.2610.07109
“LLMs are now widely used as automatic evaluators for summarization, dialogue, reasoning, and translation (Liu et al., 2023; Zheng et al., 2023).”
Self-citationYESciEval: Robust LLM-as-a-Judge for Scientific Question Answering
D'Souza, J., Giglou, H. B., & Münch, Q. (2025) · arXiv · doi:10.48550/arxiv.2505.14279
“Most prior evaluation work relies on proprietary GPT models (Wang et al., 2023a; Dubois et al., 2023; Liu et al., 2023; Fu et al., 2024), raising issues of transparency and high costs at scale (Kim et al., 2023, 2024).”
Role-Augmented Intent-Driven Generative Search Engine Optimization
Chen, X., Bao, J., Wu, H., Chen, Z., Pan, L., & Liao, Y. (2025) · arXiv · doi:10.48550/arxiv.2508.11158
“Although GEO originally used G-EVAL [15] to simulate human subjective judgment, its evaluation prompts demonstrate inconsistent scoring granularity and ambiguous criteria across dimensions, which may affect the stability of cross-method comparisons.”
Agreement Metrics for LLM-as-Judge Evaluation: What to Report and Why
Rao, D., & Callison-Burch, C. (2026) · arXiv · doi:10.48550/arxiv.2606.00093
Song, M., Kim, C., Eo, S., & Park, C. (2026) · arXiv · doi:10.48550/arxiv.2608.30373
Chen, Y., Zhang, C., Luo, D., D'Haro, L. F., Tan, R. T., & Li, H. (2024) · arXiv · doi:10.48550/arxiv.2405.14646
Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning
Seo, M., Baek, J., Lee, S., & Hwang, S. J. (2025) · arXiv · doi:10.48550/arxiv.2504.17192
Judge Reliability Harness: Stress Testing the Reliability of LLM Judges
Dev, S., Sloan, A., Kavner, J., Kong, N., & Sandler, M. (2026) · arXiv · doi:10.48550/arxiv.2603.05399
- Authors
- Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C.
- Affiliations
- Microsoft (United States) (US)
- Venue
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Published
- 2023
- Pages
- 2511-2522
- OpenAlex
- W4389519254
- Cited by
- 948
- References
- 38
- Citations by year
- 2026: 441 · 2025: 295 · 2024: 129 · 2023: 83
- FWCI
- 167.97
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- natural language generation evaluation, LLM-based evaluators, chain-of-thought prompting, GPT-4, text summarization, dialogue generation, human evaluation correlation, evaluator bias, reference-free metrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
