Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation
Open accessAbstract
New Natural Langauge Process~(NLP) benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present Xiezhi, the most comprehensive evaluation suite designed to assess holistic domain knowledge.Xiezhi comprises multiple-choice questions across 516 diverse disciplines ranging from 13 different subjects with 249,587 questions and accompanied by Xiezhi-Specialty with 14,041 questions and Xiezhi-Interdiscipline with 10,746 questions. We conduct evaluation of the 47 cutting-edge LLMs on Xiezhi. Results indicate that LLMs exceed average performance of humans in science, engineering, agronomy, medicine, and art, but fall short in economics, jurisprudence, pedagogy, literature, history, and management. All the evaluation code and data are open sourced in https://github.com/MikeGu721/XiezhiBenchmark
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
FLUE focused on the financial domain, utilizing primarily English datasets without addressing or incorporating multilingual contexts or diverse cognitive frameworks.
“The study focused on developing and evaluating language models specifically for the financial domain, utilizing primarily English datasets without addressing or incorporating multilingual contexts or the diverse cognitive frameworks shaped by different languages' unique structures.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [44]
FLUE introduced benchmarks for evaluating LLMs in the financial domain, necessitating extensive testing and iterations across various scenarios, likely extending the time frame for benchmark completion.
“The study introduced benchmarks for evaluating Large Language Models in the financial domain, necessitating extensive testing and iterations across various scenarios, likely extending the time frame for benchmark completion.The study did not explicitly acknowledge this inadequacy or mention attempts to address it.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [44]
FLUE introduced the Financial Language Understanding Evaluation (FLUE) and a new financial language model (FLANG), focusing on domain-specific pre-training and evaluation without explicitly addressing or attempting to standardize the implementation process across different teams or laboratories.
“The study introduced the Financial Language Understanding Evaluation (FLUE) and a new financial language model (FLANG), focusing on domain-specific pre-training and evaluation without explicitly addressing or attempting to standardize the implementation process across different teams or laboratories, which could lead to inconsistent benchmark applications and results.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [44]
FLUE benchmark development focused on creating assessments across various NLP tasks in the financial domain, but did not explicitly mention efforts to mitigate or acknowledge the intricacies involved in crafting unbiased and effective prompts to accurately gauge LLM capabilities.
“The benchmark development focused on creating assessments across various NLP tasks in the financial domain, but did not explicitly mention efforts to mitigate or acknowledge the intricacies involved in crafting unbiased and effective prompts to accurately gauge LLM capabilities.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [44]
FLUE introduced financial domain benchmarks without detailed consideration of cultural and ideological diversity in standardized evaluation metrics, which could conflict with values like diversity and inclusivity.
“The study introduced benchmarks in the financial domain without detailed consideration of cultural and ideological diversity in its standardized evaluation metrics, which could conflict with values like diversity and inclusivity, particularly in tasks such as sentiment analysis and news classification that inherently required cultural sensitivity.”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [44]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 31 · Cited by 28
Research works printed but not linked: 1
- Hoffmann, J.; Borgeaud, S.; Mensch, A.; Buchatskaya, E.; Cai, T.; Rutherford, E.; de Las Casas, D.; Hendricks, L. A.; Welbl, J.; Clark, A.; et al. 2022. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 35: 30016-30030.
- Authors
- Gu, Z., Zhu, X., Ye, H., Zhang, L., Wang, J., Zhu, Y., Jiang, S., Xiong, Z., Li, Z., Wu, W., He, Q., Xu, R., Huang, W., Liu, J., Wang, Z., Wang, S., Zheng, W., Feng, H., & Xiao, Y.
- Affiliations
- Fudan University (CN)
- East China University of Science and Technology (CN)
- Xiaohongshu (Little Red Book) (China) (CN)
- ORCID
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2024-03-24
- Volume
- 38
- Issue
- 16
- Pages
- 18099-18107
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4393147120
- Cited by
- 28
- References
- 31
- Citations by year
- 2026: 4 · 2025: 17 · 2024: 7
- FWCI
- 8.26
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- other-oa
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion, other-oa)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- Funders
- National Natural Science Foundation of China (U23A20496, 2021SHZDZX0103, 62306112, 62102095)
- Science and Technology Commission of Shanghai Municipality
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, multiple-choice questions, interdisciplinary knowledge, LLM evaluation, benchmark design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
