CoChat
Sign Up
PreprintCited by 446Natural Language Processing TechniquesTopic ModelingSentiment Analysis and Opinion Mining

GPTScore: Evaluate as You Desire

Open access
Fu, Jinlan, Ng, See-Kiong, Jiang, Zhengbao, Liu, Pengfei·2023·arXiv
DOI 10.48550/arxiv.2302.04166arXiv 2302.04166
Open this paper in CoChat — free accountOpen PDF

Abstract

Generative Artificial Intelligence (AI) has enabled the development of sophisticated models that are capable of producing high-caliber text, images, and other outputs through the utilization of large pre-trained models. Nevertheless, assessing the quality of the generation is an even more arduous task than the generation itself, and this issue has not been given adequate consideration recently. This paper proposes a novel evaluation framework, GPTScore, which utilizes the emergent abilities (e.g., zero-shot instruction) of generative pre-trained models to score generated texts. There are 19 pre-trained models explored in this paper, ranging in size from 80M (e.g., FLAN-T5-small) to 175B (e.g., GPT3). Experimental results on four text generation tasks, 22 evaluation aspects, and corresponding 37 datasets demonstrate that this approach can effectively allow us to achieve what one desires to evaluate for texts simply by natural language instructions. This nature helps us overcome several long-standing challenges in text evaluation--how to achieve customized, multi-faceted evaluation without the need for annotated samples. We make our code publicly available at https://github.com/jinlanfu/GPTScore.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2302.04166

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Several studies have adopted length-normalized sequence log-probability to measure model confidence.

    “Notably, several studies [91, 106, 359] have adopted length-normalized sequence log-probability to measure model confidence.”

    From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [91]

    Section 3.2 (page unavailable) defines GPTScore using token log-probabilities to evaluate generated text. It does not specify division by sequence length or describe the score as a measure of model confidence, so those details are not supported by this paper. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 8 · Cited by 446

Research works printed but not linked: 51

  1. Adiwardana, D., Luong, M., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., and Le, Q. V. Towards a human-like open- domain chatbot. CoRR, abs/2001.09977, 2020. URL https://arxiv.org/abs/2001.09977. · arXiv
  2. Bhandari, M., Gour, P. N., Ashfaq, A., Liu, P., and Neubig, G. Re-evaluating evaluation in text summarization. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 9347-9359. Association for Computational Linguistics, 2020. doi: 10.18653/v1/ 2020.emnlp-main.751. URL https://doi.org/10. 18653/v1/2020.emnlp-main.751. · DOI · Paper page
  3. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. CoRR, abs/2005.14165, 2020. URL https://arxiv.org/ abs/2005.14165. · arXiv
  4. Castricato, L., Havrilla, A., Matiana, S., Pieler, M., Ye, A., Yang, I., Frazier, S., and Riedl, M. O. Robust preference learning for storytelling via contrastive re- inforcement learning. CoRR, abs/2210.07792, 2022. doi: 10.48550/arXiv.2210.07792. URL https://doi. org/10.48550/arXiv.2210.07792. · DOI
  5. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Austin, J., Isard, M., Gur-Ari, G., Yin, P., Duke, T., Levskaya, A., Ghemawat, S., Dev, S., Michalewski, H., Garcia, X., Misra, V., Robinson, K., Fedus, L., Zhou, D., Ippolito, D., Luan, D., Lim, H., Zoph, B., Spiridonov, A., Sepassi, R., Dohan, D., Agrawal, S., Omernick, M., Dai, A. M., Pillai, T. S., Pel- lat, M., Lewkowycz, A., Moreira, E., Child, R., Polozov, O., Lee, K., Zhou, Z., Wang, X., Saeta, B., Diaz, M., Firat, O., Catasta, M., Wei, J., Meier-Hellstern, K., Eck, D., Dean, J., Petrov, S., and Fiedel, N. Palm: Scaling lan- guage modeling with pathways. CoRR, abs/2204.02311, 2022. doi: 10.48550/arXiv.2204.02311. URL https: //doi.org/10.48550/arXiv.2204.02311. · DOI · Paper page
  6. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. · arXiv
  7. Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre-training of deep bidirectional transformers for lan- guage understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171-4186. Association for Computational Linguistics, 2019. doi: 10.18653/v1/n19-1423. URL https://doi.org/ 10.18653/v1/n19-1423. · DOI · Paper page
  8. Fabbri, A. R., Kryscinski, W., McCann, B., Xiong, C., Socher, R., and Radev, D. R. Summeval: Re- evaluating summarization evaluation. Trans. Assoc. Com- put. Linguistics, 9:391-409, 2021. doi: 10.1162/tacl\ _a\_00373. URL https://doi.org/10.1162/ tacl_a_00373. · DOI · Paper page
  9. Freitag, M., Foster, G. F., Grangier, D., Ratnakar, V., Tan, Q., and Macherey, W. Experts, errors, and context: A large- scale study of human evaluation for machine translation. CoRR, abs/2104.14478, 2021. URL https://arxiv. org/abs/2104.14478. · arXiv
  10. Fu, J., Ng, S.-K., and Liu, P. Polyglot prompt: Mul- tilingual multitask promptraining. arXiv preprint arXiv:2204.14264, 2022. · arXiv
  11. Grusky, M., Naaman, M., and Artzi, Y. Newsroom: A dataset of 1.3 million summaries with diverse extrac- tive strategies. In Walker, M. A., Ji, H., and Stent, A. (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pp. 708-719. As- sociation for Computational Linguistics, 2018. doi: 10.18653/v1/n18-1065. URL https://doi.org/ 10.18653/v1/n18-1065. · DOI
  12. Hermann, K. M., Kociský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. Teaching machines to read and comprehend. In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 1693- 1701, 2015. URL https://proceedings. neurips.cc/paper/2015/hash/ afdec7005cc9f14302cd0474fd0f3c96-Abstract. html.
  13. Hu, J. E., Singh, A., Holzenberger, N., Post, M., and Durme, B. V. Large-scale, diverse, paraphrastic bitexts via sam- pling and clustering. In Bansal, M. and Villavicencio, A. (eds.), Proceedings of the 23rd Conference on Com- putational Natural Language Learning, CoNLL 2019, Hong Kong, China, November 3-4, 2019, pp. 44-54. Association for Computational Linguistics, 2019. doi: 10.18653/v1/K19-1005. URL https://doi.org/ 10.18653/v1/K19-1005. · DOI
  14. Kusner, M. J., Sun, Y., Kolkin, N. I., and Weinberger, K. Q. From word embeddings to document distances. In Bach, F. R. and Blei, D. M. (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 957-966. JMLR.org, 2015. URL http://proceedings.mlr. press/v37/kusnerb15.html.
  15. Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mo- hamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. R. (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguis- tics, ACL 2020, Online, July 5-10, 2020, pp. 7871-7880. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.acl-main.703. URL https://doi. org/10.18653/v1/2020.acl-main.703. · DOI · Paper page
  16. Li, Z., Zhang, J., Fei, Z., Feng, Y., and Zhou, J. Conversa- tions are not flat: Modeling the dynamic information flow across dialogue utterances. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 128- 138. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.acl-long.11. URL https:// doi.org/10.18653/v1/2021.acl-long. · DOI
  17. Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004. · DOI
  18. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586, 2021. · arXiv
  19. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019. URL http://arxiv.org/ abs/1907.11692. · arXiv
  20. Liu, Y., Fabbri, A. R., Liu, P., Zhao, Y., Nan, L., Han, R., Han, S., Joty, S., Wu, C.-S., Xiong, C., et al. Re- visiting the gold standard: Grounding summarization evaluation with robust human evaluation. arXiv preprint arXiv:2212.07981, 2022. · arXiv
  21. Mairesse, F., Gasic, M., Jurcícek, F., Keizer, S., Thom- son, B., Yu, K., and Young, S. J. Phrase-based statis- tical language generation using graphical models and active learning. In Hajic, J., Carberry, S., and Clark, S. (eds.), ACL 2010, Proceedings of the 48th Annual Meeting of the Association for Computational Linguis- tics, July 11-16, 2010, Uppsala, Sweden, pp. 1552-1561. The Association for Computer Linguistics, 2010. URL https://aclanthology.org/P10-1157/.
  22. Matiana, S., Smith, J. R., Teehan, R., Castricato, L., Biderman, S., Gao, L., and Frazier, S. Cut the CARP: fishing for zero-shot story evaluation. CoRR, abs/2110.03111, 2021. URL https://arxiv.org/ abs/2110.03111. · arXiv
  23. Mehri, S. and Eskénazi, M. Unsupervised evaluation of interactive dialog with dialogpt. In Pietquin, O., Mure- san, S., Chen, V., Kennington, C., Vandyke, D., Dethlefs, N., Inoue, K., Ekstedt, E., and Ultes, S. (eds.), Proceed- ings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGdial 2020, 1st virtual meeting, July 1-3, 2020, pp. 225-235. Associa- tion for Computational Linguistics, 2020. URL https: //aclanthology.org/2020.sigdial-1.28/.
  24. Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? CoRR, abs/2202.12837, 2022. URL https://arxiv. org/abs/2202.12837. · arXiv
  25. Mukaka, M. M. A guide to appropriate use of correlation coefficient in medical research. Malawi medical journal, 24(3):69-71, 2012.
  26. Nenkova, A. and Passonneau, R. Evaluating content se- lection in summarization: The pyramid method. In Pro- ceedings of the Human Language Technology Confer- ence of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pp. 145-152, Boston, Massachusetts, USA, May 2 -May 7 2004. Association for Computational Linguistics. URL https://aclanthology.org/N04-1019.
  27. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to fol- low instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. · arXiv
  28. Pang, B., Nijkamp, E., Han, W., Zhou, L., Liu, Y., and Tu, K. Towards holistic and automatic evaluation of open-domain dialogue generation. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. R. (eds.), Pro- ceedings of the 58th Annual Meeting of the Associ- ation for Computational Linguistics, ACL 2020, On- line, July 5-10, 2020, pp. 3619-3629. Association for Computational Linguistics, 2020. doi: 10.18653/v1/ 2020.acl-main.333. URL https://doi.org/10. 18653/v1/2020.acl-main.333. · DOI · Paper page
  29. Papineni, K., Roukos, S., Ward, T., and Zhu, W. Bleu: a method for automatic evaluation of machine transla- tion. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6- 12, 2002, Philadelphia, PA, USA, pp. 311-318. ACL, 2002. doi: 10.3115/1073083.1073135. URL https: //aclanthology.org/P02-1040/. · DOI · Paper page
  30. Popovic, M. chrf: character n-gram f-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, WMT@EMNLP 2015, 17-18 September 2015, Lisbon, Portugal, pp. 392-395. The Association for Computer Linguistics, 2015. doi: 10.18653/v1/w15-3049. URL https://doi.org/ 10.18653/v1/w15-3049. · DOI
  31. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  32. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-140:67, 2020. URL http://jmlr.org/papers/v21/20-074. html.
  33. Rei, R., Stewart, C., Farinha, A. C., and Lavie, A. COMET: A neural framework for MT evaluation. CoRR, abs/2009.09025, 2020. URL https://arxiv.org/ abs/2009.09025. · arXiv
  34. Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021. · arXiv
  35. Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilic, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., Tow, J., Rush, A. M., Biderman, S., Webson, A., Am- manamanchi, P. S., Wang, T., Sagot, B., Muennighoff, N., del Moral, A. V., Ruwase, O., Bawden, R., Bek- man, S., McMillan-Major, A., Beltagy, I., Nguyen, H., Saulnier, L., Tan, S., Suarez, P. O., Sanh, V., Laurençon, H., Jernite, Y., Launay, J., Mitchell, M., Raffel, C., Gokaslan, A., Simhi, A., Soroa, A., Aji, A. F., Alfassy, A., Rogers, A., Nitzav, A. K., Xu, C., Mou, C., Emezue, C., Klamm, C., Leong, C., van Strien, D., Adelani, D. I., and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022. doi: 10.48550/arXiv.2211.05100. URL https: //doi.org/10.48550/arXiv.2211.05100. · DOI · Paper page
  36. Scialom, T., Dray, P., Lamprier, S., Piwowarski, B., Sta- iano, J., Wang, A., and Gallinari, P. Questeval: Sum- marization asks for fact-based evaluation. In Moens, M., Huang, X., Specia, L., and Yih, S. W. (eds.), Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7- 11 November, 2021, pp. 6594-6604. Association for Computational Linguistics, 2021. doi: 10.18653/v1/ 2021.emnlp-main.529. URL https://doi.org/10. 18653/v1/2021.emnlp-main.529. · DOI · Paper page
  37. Sellam, T., Das, D., and Parikh, A. P. BLEURT: learn- ing robust metrics for text generation. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. R. (eds.), Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, ACL 2020, On- line, July 5-10, 2020, pp. 7881-7892. Association for Computational Linguistics, 2020. doi: 10.18653/v1/ 2020.acl-main.704. URL https://doi.org/10. 18653/v1/2020.acl-main.704. · DOI · Paper page
  38. Thompson, B. and Post, M. Automatic machine transla- tion evaluation in many languages via zero-shot para- phrasing. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 90-121. As- sociation for Computational Linguistics, 2020. doi: 10.18653/v1/2020.emnlp-main.8. URL https://doi. org/10.18653/v1/2020.emnlp-main. · DOI · Paper page
  39. Wang, A., Cho, K., and Lewis, M. Asking and answer- ing questions to evaluate the factual consistency of sum- maries. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J. R. (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguis- tics, ACL 2020, Online, July 5-10, 2020, pp. 5008-5020. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.acl-main.450. URL https://doi. org/10.18653/v1/2020.acl-main.450. · DOI
  40. Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Arunkumar, A., Ashok, A., Dhanasekaran, A. S., Naik, A., Stap, D., et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. URL https://arxiv. org/abs/2204.07705, 2022.
  41. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E. H., Le, Q., and Zhou, D. Chain of thought prompt- ing elicits reasoning in large language models. CoRR, abs/2201.11903, 2022. URL https://arxiv.org/ abs/2201.11903. · arXiv
  42. Wen, T., Gasic, M., Mrksic, N., Su, P., Vandyke, D., and Young, S. J. Semantically conditioned lstm-based nat- ural language generation for spoken dialogue systems. In Màrquez, L., Callison-Burch, C., Su, J., Pighin, D., and Marton, Y. (eds.), Proceedings of the 2015 Confer- ence on Empirical Methods in Natural Language Process- ing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp. 1711-1721. The Association for Computational Linguistics, 2015. doi: 10.18653/v1/d15-1199. URL https://doi.org/10.18653/v1/d15-1199. · DOI
  43. Xu, W., Qian, X., Wang, M., Li, L., and Wang, W. Y. Sescore2: Retrieval augmented pretraining for text gener- ation evaluation. CoRR, abs/2212.09305, 2022a. doi: 10. 48550/arXiv.2212.09305. URL https://doi.org/ 10.48550/arXiv.2212.09305. · DOI
  44. Xu, W., Tuan, Y., Lu, Y., Saxon, M., Li, L., and Wang, W. Y. Not all errors are equal: Learning text generation metrics using stratified error synthesis. In Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Findings of the As- sociation for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 6559-6574. Association for Computational Linguis- tics, 2022b. URL https://aclanthology.org/ 2022.findings-emnlp.489.
  45. Yuan, W., Neubig, G., and Liu, P. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263-27277, 2021.
  46. Zhang, C., Chen, Y., D'Haro, L. F., Zhang, Y., Friedrichs, T., Lee, G., and Li, H. Dynaeval: Unifying turn and dia- logue level evaluation. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 5676-5689. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.acl-long.441. URL https://doi. org/10.18653/v1/2021.acl-long.441. · DOI
  47. Zhang, C., D'Haro, L. F., Zhang, Q., Friedrichs, T., and Li, H. Fined-eval: Fine-grained automatic dialogue-level evaluation. CoRR, abs/2210.13832, 2022a. doi: 10. 48550/arXiv.2210.13832. URL https://doi.org/ 10.48550/arXiv.2210.13832. · DOI
  48. Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022b. · arXiv
  49. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: //openreview.net/forum?id=SkeHuCVFDr.
  50. Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C. M., and Eger, S. Moverscore: Text generation evaluating with contextualized embeddings and earth mover dis- tance. In Inui, K., Jiang, J., Ng, V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 563-578. Association for Computational Linguistics, 2019. doi: 10.18653/v1/D19-1053. URL https://doi.org/10.18653/v1/D19-1053. · DOI · Paper page
  51. Zhong, M., Liu, Y., Yin, D., Mao, Y., Jiao, Y., Liu, P., Zhu, C., Ji, H., and Han, J. Towards a uni- fied multi-dimensional evaluator for text generation. CoRR, abs/2210.07197, 2022. doi: 10.48550/arXiv. 2210.07197. URL https://doi.org/10.48550/ arXiv.2210.07197. Avg. 27.2 27.1 29.7 †, ‡ 11.3 10.4 16.4 †, ‡ 30.3 31.2 † 32.3 †, ‡ GPT2 GPT2-M 25.7 24.6 29.6 †, ‡ 8.6 9.4 † 15.1 †, ‡ 32.1 29.4 34.1 †, ‡ GPT2-L 27.2 28.5 † 32.2 †, ‡ 11.1 10.4 14.9 †, ‡ 31.2 30.9 33.9 †, ‡ GPT2-XL 24.2 27.6 † 29.7 †, ‡ 9.4 12.0 † 17.4 †, ‡ 28.6 32.2 † 35.8 †, ‡ GPT-J-6B 26.2 27.2 † 29.5 †, ‡ 9.9 11.2 † 15.9 †, ‡ 28.5 28.8 † 30.3 †, ‡ Avg. 25.8 27.0 † 30.3 †, ‡ 9.8 10.8 † 15.8 †, ‡ 30.1 30.3 † 33.5 †, ‡ OPT OPT-350M 29.3 28.1 28.6 ‡ 11.7 11.9 15.7 †, ‡ 31.5 32.5 † 31.8 OPT-1.3B 27.9 27.7 28.0 ‡ 8.8 13.3 † 15.9 †, ‡ 32.6 33.6 † 32.9 † OPT-6.7B 29.6 30.7 † 30.6 † 10.7 12.2 † 15.0 †, ‡ 34.2 36.4 † 36.9 †, ‡ OPT-13B 27.5 29.5 † 30.8 †, ‡ 9.6 11.7 † 17.9 †, ‡ 31.9 35.5 † 37.5 †, ‡ OPT-66B 29.5 31.0 † 33.4 †, ‡ 9.1 12.1 † 16.8 †, ‡ 32.1 35.3 † 36.4 †, ‡ Avg. 28.7 29.4 † 30.3 †, ‡ 10.0 12.2 † 16.3 †, ‡ 32.5 34.6 † 35.1 †, ‡ FLAN-T5 FT5-S 27.6 28.7 † 27.0 12.6 9.4 15.0 †, ‡ 33.5 33.3 31.3 FT5-B 25.5 25.4 27.4 †, ‡ 10.4 10.2 15.9 †, ‡ 29.8 29.6 30.0 ‡ FT5-L 28.5 28.5 28.8 †, ‡ 7.9 13.0 † 15.6 †, ‡ 30.7 31.6 † 32.1 †, ‡ FT5-XL 28.1 27.0 28.1 ‡ 9.4 10.2 † 14.0 †, ‡ 30.4 33.5 † 34.2 †, ‡ FT5-XXL 29.0 29.4 † 30.5 †, ‡ 7.6 12.2 † 16.2 †, ‡ 30.7 33.3 † 33.8 †, ‡ Avg. 27.7 27.8 28.3 †, ‡ 9.6 11.0 † 15.4 †, ‡ 31.0 32.3 † 32.3 † Overall Avg 27.4 27.8 † 29.7 †, ‡ 10.2 11.1 † 16.0 †, ‡ 31.0 32.1 † 33.3 †, ‡ Table · DOI · Paper page

Authors
Fu, J., Ng, S.-K., Jiang, Z., & Liu, P.
Affiliations
  • National University of Singapore (SG)
  • Carnegie Mellon University (US)
  • Shanghai Jiao Tong University (CN)
Venue
arXiv
Publisher
arXiv
Published
2023
Pages
6556-6576
DOI
10.48550/arxiv.2302.04166
arXiv
2302.04166
OpenAlex
W4401043863
Cited by
446
References
8
Citations by year
2026: 84 · 2025: 146 · 2024: 152 · 2023: 64
FWCI
92.33
Type
conference paper
Language
en
Open access
Licence
Locations
  • Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • National Research Foundation
  • National Research Foundation Singapore
  • Defence Science and Technology Group
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
LLM evaluation, evaluation metrics, GPT-4

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review