CoChat
Sign Up
PreprintCited by 104Topic ModelingNatural Language Processing TechniquesEthics and Social Impacts of AI

The DOI matches its DataCite record.

Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence

Open access
McIntosh, Timothy R., Susnjak, Teo, Arachchilage, Nalin, Liu, Tong, Watters, Paul, Halgamuge, Malka N.·2024·arXiv
DOI 10.48550/arxiv.2402.09880arXiv 2402.09880
Open this paper in CoChat — free accountOpen PDF

Abstract

The rapid rise in popularity of Large Language Models (LLMs) with emerging capabilities has spurred public curiosity to evaluate and compare different LLMs, leading many researchers to propose their own LLM benchmarks. Noticing preliminary inadequacies in those benchmarks, we embarked on a study to critically assess 23 state-of-the-art LLM benchmarks, using our novel unified evaluation framework through the lenses of people, process, and technology, under the pillars of benchmark functionality and integrity. Our research uncovered significant limitations, including biases, difficulties in measuring genuine reasoning, adaptability, implementation inconsistencies, prompt engineering complexity, evaluator diversity, and the overlooking of cultural and ideological norms in one comprehensive assessment. Our discussions emphasized the urgent need for standardized methodologies, regulatory certainties, and ethical guidelines in light of Artificial Intelligence (AI) advancements, including advocating for an evolution from static benchmarks to dynamic behavioral profiling to accurately capture LLMs' complex behaviors and potential risks. Our study highlighted the necessity for a paradigm shift in LLM evaluation methodologies, underlining the importance of collaborative efforts for the development of universally accepted benchmarks and the enhancement of AI systems' integration into society.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.09880

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • If a benchmark allows shortcuts, accuracy on the benchmark does not translate to the real world.

    “If a benchmark allows shortcuts, accuracy on the benchmark does not translate to the real world [32, 33, 52].”

    From AI Agents That Matter (2024) · cites this paper as [32]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 60

  1. B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, "Recent advances in natural language processing via large pre-trained language models: A survey," ACM Computing Surveys, vol. 56, no. 2, pp. 1-40, 2023.
  2. T. R. McIntosh, T. Susnjak, T. Liu, P. Watters, and M. N. Halgamuge, "From google gemini to openai q*(q-star): A survey of reshaping the generative artificial intelligence (ai) research landscape," arXiv preprint arXiv:2312.10868, 2023. · arXiv
  3. Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang et al., "A survey on evaluation of large language models," ACM Transactions on Intelligent Systems and Technology, 2023.
  4. T. R. McIntosh, T. Liu, T. Susnjak, P. Watters, and M. N. Halgamuge, "A reasoning and value alignment test to assess advanced gpt reasoning," ACM Transactions on Interactive Intelligent Systems, 2024.
  5. R. Liu and N. Moini, "Benchmarking transportation safety performance via shift-share approaches," Journal of Transportation Safety & Security, vol. 7, no. 2, pp. 124-137, 2015.
  6. A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso et al., "Beyond the imitation game: Quantifying and extrapolating the capabilities of language models," Transactions on machine learning research, 2022.
  7. K. Zhu, Q. Zhao, H. Chen, J. Wang, and X. Xie, "Promptbench: A unified library for evaluation of large language models," Journal of Machine Learning Research, vol. 25, no. 254, pp. 1-22, 2024.
  8. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107.03374, 2021. · arXiv
  9. C. Liu, R. Jin, Y. Ren, L. Yu, T. Dong, X. Peng, S. Zhang, J. Peng, P. Zhang, Q. Lyu et al., "M3ke: A massive multi-level multi-subject knowledge evaluation benchmark for chinese large language models," arXiv preprint arXiv:2305.10263, 2023. · arXiv
  10. K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl et al., "Large language models encode clinical knowledge," Nature, vol. 620, no. 7972, pp. 172- 180, 2023.
  11. J. Yu, X. Wang, S. Tu, S. Cao, D. Zhang-Li, X. Lv, H. Peng, Z. Yao, X. Zhang, H. Li et al., "Kola: Carefully benchmarking world knowledge of large language models," arXiv preprint arXiv:2306.09296, 2023. · arXiv
  12. W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan, "Agieval: A human-centric benchmark for evaluating foundation models," in Findings of the Association for Computational Linguistics: NAACL 2024, 2024, pp. 2299-2314.
  13. T. R. McIntosh, T. Liu, T. Susnjak, P. Watters, A. Ng, and M. N. Halga- muge, "A culturally sensitive test to evaluate nuanced gpt hallucination," IEEE Transactions on Artificial Intelligence, 2023.
  14. N. Guha, J. Nyarko, D. Ho, C. Ré, A. Chilton, A. Chohlas-Wood, A. Peters, B. Waldon, D. Rockmore, D. Zambrano et al., "Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models," Advances in Neural Information Processing Systems, vol. 36, 2024.
  15. A. V. Papadopoulos, L. Versluis, A. Bauer, N. Herbst, J. Von Kistowski, A. Ali-Eldin, C. L. Abad, J. N. Amaral, P. Tuma, and A. Iosup, "Methodological principles for reproducible performance evaluation in cloud computing," IEEE Transactions on Software Engineering, vol. 47, no. 8, pp. 1528-1543, 2019.
  16. H. Brunst, S. Chandrasekaran, F. M. Ciorba, N. Hagerty, R. Henschel, G. Juckeland, J. Li, V. G. M. Vergara, S. Wienke, and M. Zavala, "First experiences in performance benchmarking with the new spechpc 2021 suites," in 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid). IEEE, 2022, pp. 675-684.
  17. B. Li, W. Ren, D. Fu, D. Tao, D. Feng, W. Zeng, and Z. Wang, "Benchmarking single-image dehazing and beyond," IEEE Transactions on Image Processing, vol. 28, no. 1, pp. 492-505, 2018.
  18. M. Hort, M. Kechagia, F. Sarro, and M. Harman, "A survey of per- formance optimization for mobile applications," IEEE Transactions on Software Engineering, vol. 48, no. 8, pp. 2879-2904, 2021.
  19. M. S. Aslanpour, S. S. Gill, and A. N. Toosi, "Performance evaluation metrics for cloud, fog and edge computing: A review, taxonomy, bench- marks and standards for future research," Internet of Things, vol. 12, p. 100273, 2020.
  20. F. Xia, W. B. Shen, C. Li, P. Kasimbeg, M. E. Tchapmi, A. Toshev, R. Martín-Martín, and S. Savarese, "Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments," IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 713-720, 2020.
  21. A. Romano, X. Liu, Y. Kwon, and W. Wang, "An empirical study of bugs in webassembly compilers," in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2021, pp. 42-54.
  22. E. Davis, "Benchmarks for automated commonsense reasoning: A survey," ACM Computing Surveys, vol. 56, no. 4, p. 41, 2023.
  23. S. Zhu, J. Shi, L. Yang, B. Qin, Z. Zhang, L. Song, and G. Wang, "Measuring and modeling the label dynamics of online {Anti-Malware} engines," in 29th USENIX Security Symposium (USENIX Security 20), 2020, pp. 2361-2378.
  24. T. McIntosh, A. Kayes, Y.-P. P. Chen, A. Ng, and P. Watters, "Ran- somware mitigation in the modern era: A comprehensive review, re- search challenges, and future directions," ACM Computing Surveys (CSUR), vol. 54, no. 9, pp. 1-36, 2021.
  25. T. A. van Schaik and B. Pugh, "A field guide to automatic evaluation of llm-generated summaries," in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2024, pp. 2832-2836.
  26. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bow- man, "Glue: A multi-task benchmark and analysis platform for natural language understanding," in 7th International Conference on Learning Representations, ICLR 2019, 2019.
  27. Z. Tan, T. Chen, Z. Zhang, and H. Liu, "Sparsity-guided holistic explanation for llms with interpretable inference-time intervention," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 19, 2024, pp. 21 619-21 627.
  28. T. R. McIntosh, T. Susnjak, T. Liu, P. Watters, and M. N. Halgamuge, "The inadequacy of reinforcement learning from human feedback- radicalizing large language models via semantic vulnerabilities," IEEE Transactions on Cognitive and Developmental Systems, 2024.
  29. G. F. Almeida, J. L. Nunes, N. Engelmann, A. Wiegmann, and M. de Araújo, "Exploring the psychology of llms' moral and legal reasoning," Artificial Intelligence, vol. 333, p. 104145, 2024.
  30. T. R. McIntosh, T. Susnjak, T. Liu, P. Watters, D. Xu, D. Liu, R. Nowrozy, and M. N. Halgamuge, "From cobit to iso 42001: Evalu- ating cybersecurity frameworks for opportunities, risks, and regulatory compliance in commercializing large language models," Computers & Security, vol. 144, p. 103964, 2024.
  31. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar et al., "Holistic evaluation of language models," arXiv preprint arXiv:2211.09110, 2022. · arXiv
  32. K. Kenthapadi, M. Sameki, and A. Taly, "Grounding and evaluation for large language models: Practical challenges and lessons learned (survey)," in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, pp. 6523-6533.
  33. Z. Chen, H. Mao, H. Li, W. Jin, H. Wen, X. Wei, S. Wang, D. Yin, W. Fan, H. Liu et al., "Exploring the potential of large language models (llms) in learning on graphs," ACM SIGKDD Explorations Newsletter, vol. 25, no. 2, pp. 42-61, 2024.
  34. M. Jegorova, C. Kaul, C. Mayor, A. Q. O'Neil, A. Weir, R. Murray- Smith, and S. A. Tsaftaris, "Survey: Leakage and privacy at inference time," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, no. 7, pp. 9090-9108, 2022.
  35. Y. Deng, W. Zhang, S. J. Pan, and L. Bing, "Multilingual jailbreak challenges in large language models," arXiv preprint arXiv:2310.06474, 2023. · arXiv
  36. I. Cheong, K. Xia, K. K. Feng, Q. Z. Chen, and A. X. Zhang, "(a) i am not a lawyer, but...: Engaging legal experts towards responsible llm policies for legal advice," in The 2024 ACM Conference on Fairness, Accountability, and Transparency, 2024, pp. 2454-2469.
  37. M. I. Javaid and M. M. W. Iqbal, "A comprehensive people, process and technology (ppt) application model for information systems (is) risk management in small/medium enterprises (sme)," in 2017 International Conference on Communication Technologies (ComTech). IEEE, 2017, pp. 78-90.
  38. D. Al-Fraihat, M. Joy, J. Sinclair et al., "Evaluating e-learning systems success: An empirical study," Computers in human behavior, vol. 102, pp. 67-86, 2020.
  39. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, "Measuring massive multitask language understanding," arXiv preprint arXiv:2009.03300, 2020. · arXiv
  40. Y. Fu, L. Ou, M. Chen, Y. Wan, H. Peng, and T. Khot, "Chain-of-thought hub: A continuous effort to measure large language models' reasoning performance," arXiv preprint arXiv:2305.17306, 2023. · arXiv
  41. T. Sawada, D. Paleka, A. Havrilla, P. Tadepalli, P. Vidas, A. Kranias, J. J. Nay, K. Gupta, and A. Komatsuzaki, "Arb: Advanced reasoning bench- mark for large language models," arXiv preprint arXiv:2307.13692, 2023. · arXiv
  42. Z. Gu, X. Zhu, H. Ye, L. Zhang, J. Wang, S. Jiang, Z. Xiong, Z. Li, Q. He, R. Xu et al., "Xiezhi: An ever-updating benchmark for holistic domain knowledge evaluation," arXiv preprint arXiv:2306.05783, 2023. · arXiv
  43. Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, J. Lei et al., "C-eval: A multi-level multi- discipline chinese evaluation suite for foundation models," arXiv preprint arXiv:2305.08322, 2023. · arXiv
  44. R. Shah, K. Chawla, D. Eidnani, A. Shah, W. Du, S. Chava, N. Raman, C. Smiley, J. Chen, and D. Yang, "When flue meets flang: Benchmarks and large pretrained language model for financial domain," in Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 2322-2335.
  45. Q. Xu, F. Hong, B. Li, C. Hu, Z. Chen, and J. Zhang, "On the tool manipulation capability of open-sourced large language models," in NeurIPS 2023 Foundation Models for Decision Making Workshop.
  46. X. Wang, Z. Hu, P. Lu, Y. Zhu, J. Zhang, S. Subramaniam, A. R. Loomba, S. Zhang, Y. Sun, and W. Wang, "Scibench: Evaluating college- level scientific problem-solving abilities of large language models," arXiv preprint arXiv:2307.10635, 2023. · arXiv
  47. Q. Tang, Z. Deng, H. Lin, X. Han, Q. Liang, and L. Sun, "Toolalpaca: Generalized tool learning for language models with 3000 simulated cases," arXiv preprint arXiv:2306.05301, 2023. · arXiv
  48. Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian et al., "Toolllm: Facilitating large language models to master 16000+ real-world apis," arXiv preprint arXiv:2307.16789, 2023. · arXiv
  49. X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang et al., "Agentbench: Evaluating llms as agents," arXiv preprint arXiv:2308.03688, 2023. · arXiv
  50. M. Li, F. Song, B. Yu, H. Yu, Z. Li, F. Huang, and Y. Li, "Api-bank: A benchmark for tool-augmented llms," arXiv preprint arXiv:2304.08244, 2023. · arXiv
  51. Z. Liu, W. Yao, J. Zhang, L. Xue, S. Heinecke, R. Murthy, Y. Feng, Z. Chen, J. C. Niebles, D. Arpit et al., "Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents," arXiv preprint arXiv:2308.05960, 2023. · arXiv
  52. J. Li, X. Cheng, W. X. Zhao, J.-Y. Nie, and J.-R. Wen, "Halueval: A large-scale hallucination evaluation benchmark for large language models," in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 6449-6464.
  53. A. Wei, N. Haghtalab, and J. Steinhardt, "Jailbroken: How does llm safety training fail?" in Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  54. S. Balloccu, P. Schmidtová, M. Lango, and O. Dušek, "Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-source llms," arXiv preprint arXiv:2402.03927, 2024. · arXiv
  55. K. Zhou, Y. Zhu, Z. Chen, W. Chen, W. X. Zhao, X. Chen, Y. Lin, J.-R. Wen, and J. Han, "Don't make your llm an evaluation benchmark cheater," arXiv preprint arXiv:2311.01964, 2023. · arXiv
  56. L. Perlovsky, "Language and emotions: emotional sapir-whorf hypoth- esis," Neural Networks, vol. 22, no. 5-6, pp. 518-526, 2009.
  57. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, "Survey of hallucination in natural language generation," ACM Computing Surveys, vol. 55, no. 12, pp. 1-38, 2023.
  58. K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, "Not what you've signed up for: Compromising real-world llm- integrated applications with indirect prompt injection," in Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, 2023, pp. 79-90.
  59. J. Zamfirescu-Pereira, R. Y. Wong, B. Hartmann, and Q. Yang, "Why johnny can't prompt: how non-ai experts try (and fail) to design llm prompts," in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 2023, pp. 1-21.
  60. L. Yu, Y. Leng, Y. Huang, S. Wu, H. Liu, X. Ji, J. Zhao, J. Song, T. Cui, X. Cheng et al., "Cmoraleval: A moral evaluation benchmark for chinese large language models," in Findings of the Association for Computational Linguistics ACL 2024, 2024, pp. 11 817-11 837.

Cited by

Sorted by citations

Metadata

Authors
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Watters, P., & Halgamuge, M. N.
Affiliations
  • RMIT University (AU)
  • Massey University (NZ)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Volume
7
Issue
1
Pages
22-39
ISSN
2691-4581 (ISSN-L)
DOI
10.48550/arxiv.2402.09880
arXiv
2402.09880
OpenAlex
W4410342726
Cited by
104
References
0
Citations by year
2026: 25 · 2025: 32 · 2024: 46
FWCI
22.86
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
Keywords
large language models, LLM benchmark, benchmark evaluation, benchmark bias, reasoning evaluation, prompt engineering, AI ethics, AI regulation, benchmark standardization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review