CoChat
Sign Up
PreprintCited by 21

C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models

Huang, Yuzhen, Bai, Yuzhuo, Zhu, Zhihao, Zhang, Junlei, Zhang, Jinghan, Su, Tangjun, Liu, Junteng, Lv, Chuancheng +5 more·2023·arXiv
DOI 10.48550/arxiv.2305.08322arXiv 2305.08322
Open this paper in CoChat — free account

Abstract

New NLP benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present C-Eval, the first comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. C-Eval comprises multiple-choice questions across four difficulty levels: middle school, high school, college, and professional. The questions span 52 diverse disciplines, ranging from humanities to science and engineering. C-Eval is accompanied by C-Eval Hard, a subset of very challenging subjects in C-Eval that requires advanced reasoning abilities to solve. We conduct a comprehensive evaluation of the most advanced LLMs on C-Eval, including both English- and Chinese-oriented models. Results indicate that only GPT-4 could achieve an average accuracy of over 60%, suggesting that there is still significant room for improvement for current LLMs. We anticipate C-Eval will help analyze important strengths and shortcomings of foundation models, and foster their development and growth for Chinese users.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.08322

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • C-Eval is the first extensive benchmark to assess advanced knowledge and reasoning capabilities of foundation models in Chinese.

    “C-Eval [78], which is the first extensive benchmark to assess the advanced knowledge and reasoning capabilities of foundation models in Chinese.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [78]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 37

  1. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. · arXiv
  2. Ondřej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Level- ing, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. Findings of the 2014 workshop on statistical machine translation. In Proceed- ings of the Ninth Workshop on Statistical Machine Translation, pp. 12-58, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. doi: 10.3115/v1/W14-3302. URL https://aclanthology.org/W14-3302. · DOI
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. · arXiv
  4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. · arXiv
  5. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. · arXiv
  6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. · arXiv
  7. Yiming Cui, Ziqing Yang, and Xin Yao. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177, 2023. URL https://arxiv.org/abs/2304.08177. · arXiv
  8. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320-335, 2022.
  9. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. arXiv preprint arXiv:2211.10435, 2022. · arXiv
  10. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a. URL https://openreview.net/forum?id=d7KBjmI3GmQ.
  11. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Infor- mation Processing Systems Track on Datasets and Benchmarks, volume 1. Curran, 2021b. URL https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/ 2021/file/be83ab3ecd0db773eb2dc1b0a17836a1-Paper-round2.pdf.
  12. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
  13. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. · arXiv
  14. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pp. 4171- 4186, 2019.
  15. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=e2TBb5y0yFf.
  16. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. · arXiv
  17. Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842, 2023. · arXiv
  18. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786, 2022. · arXiv
  19. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ D18-1206. URL https://aclanthology.org/D18-1206. · DOI · Paper page
  20. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474, 2022. · arXiv
  21. OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. · arXiv
  22. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  23. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, 2018.
  24. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. · arXiv
  25. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. · arXiv
  26. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022. · arXiv
  27. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022. · arXiv
  28. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. · arXiv
  29. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. URL https://openreview.net/ forum?id=rJ4km2R5t7.
  30. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=1PL1NIMMrw.
  31. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022. · arXiv
  32. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id= _VjQlMeSB_J.
  33. Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie. Decom- position enhances reasoning via self-evaluation guided decoding. arXiv preprint arXiv:2305.00633, 2023. · arXiv
  34. Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. CLUE: A Chinese language understanding evaluation benchmark. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 4762- 4772, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10.18653/v1/2020.coling-main.419. URL https://aclanthology.org/2020. coling-main.419. · DOI
  35. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=-Aw0rrrPUF.
  36. Hui Zeng. Measuring massive multitask chinese understanding. arXiv preprint arXiv:2304.12986, 2023. · arXiv
  37. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=WZH7099tgfM. Subject GPT-4 ChatGPT ChatGLM Claude-instant-v1.0 Advanced Mathematics 49.7 / 53.8 48.0 / 29.5 3.5 / 23.1 39.9 / 41.6 College Chemistry 59.4 / 55.8 39.7 / 33.9 17.0 / 28.6 33.5 / 32.1 College Physics 50.6 / 60.2 35.8 / 35.2 14.8 / 30.1 35.8 / 31.3 College Programming 78.1 / 77.5 57.9 / 57.9 18.7 / 24.0 48.5 / 50.3 Computer Architecture 74.6 / 75.6 68.9 / 63.7 30.1 / 27.5 52.3 / 50.8 Computer Network 77.8 / 77.2 58.5 / 59.6 36.3 / 28.1 46.8 / 53.8 Discrete Mathematics 66.7 / 62.7 43.1 / 30.7 17.0 / 26.1 32.7 / 33.3 Electrical Engineer 49.9 / 56.0 39.5 / 44.0 23.9 / 31.3 31.3 / 35.7 High School Biology 72.6 / 70.3 53.7 / 48.0 28.0 / 26.9 44.6 / 38.9 High School Chemistry 52.3 / 55.8 52.9 / 36.0 19.8 / 28.5 39.0 / 37.8 High School Mathematics 38.0 / 42.8 34.3 / 37.3 4.2 / 24.1 24.1 / 32.5 High School Physics 69.1 / 61.1 43.4 / 34.9 10.9 / 21.7 44.0 / 26.9 Metrology Engineer 70.3 / 71.2 63.0 / 59.8 41.1 / 48.4 50.7 / 55.3 Middle School Biology 91.2 / 86.5 67.7 / 63.5 30.7 / 38.0 55.2 / 57.3 Middle School Chemistry 81.1 / 71.4 71.4 / 60.0 36.8 / 42.2 58.4 / 59.5 Middle School Mathematics 62.7 / 66.7 44.6 / 42.4 3.4 / 20.3 28.8 / 33.3 Middle School Physics 81.5 / 78.7 66.9 / 57.9 20.2 / 41.0 52.8 / 53.4 Operating System 82.1 / 80.4 71.5 / 60.3 40.8 / 27.4 57.0 / 57.5 Probability and Statistics 53.6 / 62.0 33.7 / 42.8 27.7 / 26.5 34.9 / 31.9 Veterinary Medicine 80.0 / 80.0 64.3 / 58.1 39.5 / 35.2 51.0 / 53.3 Business Administration 67.1 / 65.4 52.8 / 46.8 32.6 / 34.9 44.2 / 42.9 College Economics 71.4 / 74.4 52.5 / 51.3 22.1 / 30.0 47.7 / 50.3 Education Science 68.5 / 69.3 59.6 / 54.4 37.8 / 33.0 51.1 / 51.9 High School Geography 74.7 / 70.2 58.4 / 55.1 29.2 / 38.2 52.8 / 49.4 High School Politics 77.8 / 69.3 52.3 / 51.1 25.0 / 29.6 38.1 / 39.8 Mao Zedong Thought 79.5 / 79.9 60.7 / 63.5 46.6 / 50.7 47.0 / 50.2 Marxism 81.6 / 82.7 71.5 / 70.4 40.2 / 49.2 66.5 / 63.7 Middle School Geography 83.3 / 83.3 60.2 / 56.5 33.3 / 44.4 62.0 / 52.8 Middle School Politics 87.1 / 85.5 74.1 / 67.4 41.5 / 40.9 60.6 / 64.3 Teacher Qualification 85.0 / 85.0 75.7 / 66.9 48.9 / 49.1 68.4 / 62.2 Art Studies 65.1 / 66.1 49.7 / 52.3 34.6 / 36.9 35.6 / 38.9 Chinese Language and Literature 61.2 / 67.0 50.2 / 51.7 32.5 / 37.3 41.2 / 44.0 High School Chinese 37.6 / 39.3 36.0 / 27.5 26.4 / 19.1 30.9 / 27.0 High School History 68.1 / 68.1 54.4 / 45.1 35.2 / 40.1 52.7 / 40.7 Ideological and Moral Cultivation 77.3 / 77.3 66.9 / 68.0 48.3 / 52.3 62.8 / 63.4 Law 60.6 / 54.8 43.9 / 34.8 23.5 / 29.9 38.0 / 33.0 Legal Professional 54.4 / 48.4 44.7 / 32.6 27.0 / 26.5 39.5 / 30.2 Logic 60.3 / 63.2 37.7 / 35.8 32.8 / 31.4 38.7 / 35.8 Middle School History 84.5 / 86.5 62.8 / 63.8 48.3 / 52.7 58.5 / 52.2 Modern Chinese History 68.9 / 67.0 52.8 / 51.4 36.3 / 39.2 44.8 / 44.8 Professional Tour Guide 71.8 / 70.3 61.3 / 62.0 44.0 / 51.9 43.6 / 44.0 Accountant 64.6 / 61.9 51.7 / 42.0 23.9 / 32.7 47.2 / 41.3 Basic Medicine 80.6 / 78.3 61.1 / 60.6 31.4 / 33.7 49.7 / 45.1 Civil Servant 62.5 / 59.0 46.9 / 43.1 24.5 / 29.6 42.0 / 41.3 Clinical Medicine 76.0 / 73.5 55.0 / 53.0 34.5 / 32.0 36.5 / 37.5 Environmental Impact Assessment Engineer 63.7 / 59.8 50.9 / 49.1 37.7 / 35.6 47.3 / 47.3 Fire Engineer 50.0 / 49.6 42.6 / 37.9 28.4 / 32.3 36.5 / 36.2 Physician 76.8 / 76.1 63.7 / 54.6 34.8 / 37.7 50.6 / 46.5 Plant Protection 78.9 / 80.4 65.8 / 57.8 43.2 / 36.2 56.3 / 50.3 Sports Science 72.2 / 70.0 58.3 / 50.6 42.2 / 41.1 45.6 / 46.7 Tax Accountant 58.0 / 58.2 42.0 / 35.7 16.3 / 30.9 42.7 / 31.8 Urban and Rural Planner 63.2 / 65.6 52.2 / 49.3 36.8 / 37.3 45.5 / 42.3

Cited by

Sorted by citations

Authors
Huang, Y., Bai, Y., Zhu, Z., Zhang, J., Zhang, J., Su, T., Liu, J., Lv, C., Zhang, Y., Lei, J., Fu, Y., Sun, M., & He, J.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2305.08322
arXiv
2305.08322
References
1
Type
Preprint
Licence
creativecommons.org/licenses/by-nc-sa/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review