Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Evaluating the quality of generated text is a challenging task in NLP, due to the inherent complexity and diversity of text.Recently, large language models (LLMs) have garnered significant attention due to their impressive performance in various tasks.Therefore, we present this paper to investigate the effectiveness of LLMs, especially ChatGPT, and explore ways to optimize their use in assessing text quality.We compared three kinds of referencefree evaluation methods.The experimental results prove that ChatGPT is capable of evaluating text quality effectively from various perspectives without reference and demonstrates superior performance than most existing automatic metrics.In particular, the Explicit Score, which utilizes ChatGPT to generate a numeric score measuring text quality, is the most effective and reliable method among the three exploited approaches.However, directly comparing the quality of two texts may lead to suboptimal results.We believe this paper will provide valuable insights for evaluating text quality with LLMs and have released the used data 1 .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
SimCSE: Simple Contrastive Learning of Sentence Embeddings
Gao, T., Yao, X., & Chen, D. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.552 · Cited by 2,851
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · cited 1×
“Nowadays, large language models (LLMs) (Brown et al., 2020; Ouyang et al., 2022; Chung et al., 2022; Chowdhery et al., 2022; Zhang et al., 2022; Touvron et al., 2023; Du et al., 2022) represented by ChatGPT 2 have revolutionized the field of NLP by achieving remarkable results in a wide range of…”
BackgroundSummEval: Re-evaluating Summarization Evaluation
Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00373 · Cited by 587 · cited 2×
“Furthermore, different types of text necessitate evaluation of distinct aspects, e.g. coherence, fluency, and consistency (Fabbri et al., 2021a; Mehri and Eskenazi, 2020a; Wang et al., 2023b), which makes it hard to design metrics for each type of text and dimension separately.”
BackgroundMoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C. M., & Eger, S. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1053 · Cited by 490 · cited 2×
“…metrics from the following widely used metrics accordingly: ROUGE-1 , ROUGE-2 and ROUGE-L (Lin, 2004); BERTScore (Zhang* et al., 2020); MoverScore (Zhao et al., 2019); PRISM (Thompson and Post, 2020); BARTScore and its enhanced versions, BARTScore+CNN and BARTScore+CNN+Para (Yuan et al., 2021);…”
MethodGPTScore: Evaluate as You Desire
Fu, J., Ng, S.-K., Jiang, Z., & Liu, P. (2023) · arXiv · doi:10.48550/arxiv.2302.04166 · Cited by 446
Cited by
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
The Prompt Report: A Systematic Survey of Prompt Engineering Techniques
Schulhoff, S., Ilie, M., Balepur, N., Kahadze, K., Liu, A., Si, C., Li, Y., Gupta, A., Han, H., Schulhoff, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.06608 · Cited by 118
Benchmarking Cognitive Biases in Large Language Models as Evaluators
Koo, R., Lee, M., Raheja, V., Park, J. I., Kim, Z. M., & Kang, D. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.29 · Cited by 371
Jatmo: Prompt Injection Defense by Task-Specific Finetuning
Piet, J., Alrashed, M., Sitawarin, C., Chen, S., Wei, Z., Sun, E., Alomair, B., & Wagner, D. (2023) · arXiv · doi:10.48550/arxiv.2312.17673 · Cited by 54
“For generative tasks, we rely on automated rating by a language model, a standard approach used for evaluation [30, 36, 12, 17, 51, 24, 11, 44] known to be more accurate than traditional metrics such as perplexity.”
MethodLLM-LADE: Large language model-based log anomaly detection with explanation
Zhang, Z., Li, S., Zhang, L., Ye, J., Hu, C., & Yan, L. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114064 · Cited by 42
Metadata
- Authors
- Chen, Y., Wang, R., Jiang, H., Shi, S., & Xu, R.
- Venue
- Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings)
- Published
- 2023
- Pages
- 361-374
- OpenAlex
- W4392669905
- Cited by
- 68
- References
- 16
- FWCI
- 12.85
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study. https://app.cochat.ai/papers/10.18653/v1/2023.findings-ijcnlp.32
- The paper
- Chen, Y., Wang, R., Jiang, H., Shi, S., & Xu, R. (2023). Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study. Findings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings). https://doi.org/10.18653/v1/2023.findings-ijcnlp.32
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.