Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record.

Reasoning Runtime Behavior of a Program with LLM: How Far Are We?

Chen, Junkai, Pan, Zhiyuan, Hu, Xing, Li, Zhenhao, Li, Ge, Xia, Xin·2024·arXiv
DOI 10.48550/arxiv.2403.16437arXiv 2403.16437
Open this paper in CoChat — free account

Abstract

Large language models for code (i.e., code LLMs) have shown strong code understanding and generation capabilities. To evaluate the capabilities of code LLMs in various aspects, many benchmarks have been proposed (e.g., HumanEval and ClassEval). Code reasoning is one of the most essential abilities of code LLMs, but existing benchmarks for code reasoning are not sufficient. Typically, they focus on predicting the input and output of a program, ignoring the evaluation of the intermediate behavior during program execution, as well as the logical consistency (e.g., the model should not give the correct output if the prediction of execution path is wrong) when performing the reasoning. To address these problems, in this paper, we propose a framework, namely REval, for evaluating code reasoning abilities and consistency of code LLMs with program execution. We utilize existing code benchmarks and adapt them to new benchmarks within our framework. A large-scale empirical study is conducted and most LLMs show unsatisfactory performance on both Runtime Behavior Reasoning (i.e., an average accuracy of 44.4%) and Incremental Consistency Evaluation (i.e., an average IC score of 10.3). Evaluation results of current code LLMs reflect the urgent need for the community to strengthen the code reasoning capability of code LLMs. Our code, data, and \newname leaderboard are available at https://r-eval.github.io.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2403.16437

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 58

  1. B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, "Recent advances in natural language processing via large pre-trained language models: A survey," ACM Computing Surveys, vol. 56, no. 2, pp. 1-40, 2023.
  2. F. Barbieri, L. E. Anke, and J. Camacho-Collados, "Xlm-t: Multilingual language models in twitter for sentiment analysis and beyond," in Proceedings of the Thirteenth Language Resources and Evaluation Conference, 2022, pp. 258-266.
  3. A. Creswell, M. Shanahan, and I. Higgins, "Selection-inference: Exploit- ing large language models for interpretable logical reasoning," in The Eleventh International Conference on Learning Representations, 2022.
  4. A. Rogers, M. Gardner, and I. Augenstein, "Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension," ACM Computing Surveys, vol. 55, no. 10, pp. 1-45, 2023.
  5. A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, "Large language models for software engineering: Survey and open problems," arXiv preprint arXiv:2310.03533, 2023. · arXiv
  6. B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin et al., "Code llama: Open foundation models for code," arXiv preprint arXiv:2308.12950, 2023. · arXiv
  7. Y. Wei, Z. Wang, J. Liu, Y. Ding, and L. Zhang, "Magicoder: Source code is all you need," arXiv preprint arXiv:2312.02120, 2023. · arXiv
  8. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107.03374, 2021. · arXiv
  9. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., "Program synthesis with large language models," arXiv preprint arXiv:2108.07732, 2021. · arXiv
  10. S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang et al., "Codexglue: A machine learning benchmark dataset for code understanding and generation," in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
  11. Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang et al., "A survey on evaluation of large language models," ACM Transactions on Intelligent Systems and Technology, 2023.
  12. A. Ni, S. Iyer, D. Radev, V. Stoyanov, W.-t. Yih, S. Wang, and X. V. Lin, "Lever: Learning to verify language-to-code generation with execution," in International Conference on Machine Learning. PMLR, 2023, pp. 26 106-26 128.
  13. X. Chen, M. Lin, N. Schärli, and D. Zhou, "Teaching large language models to self-debug," arXiv preprint arXiv:2304.05128, 2023. · arXiv
  14. A. Gu, B. Rozière, H. Leather, A. Solar-Lezama, G. Synnaeve, and S. I. Wang, "Cruxeval: A benchmark for code reasoning, understanding and execution," arXiv preprint arXiv:2401.03065, 2024. · arXiv
  15. C. Liu, S. D. Zhang, and R. Jabbarvand, "Codemind: A framework to challenge large language models for code reasoning," arXiv preprint arXiv:2402.09664, 2024. · arXiv
  16. Y. Ding, B. Steenhoek, K. Pei, G. Kaiser, W. Le, and B. Ray, "Traced: Execution-aware pre-training for source code," in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1-12.
  17. Y. Peng, C. Gao, Z. Li, B. Gao, D. Lo, Q. Zhang, and M. Lyu, "Static inference meets deep learning: a hybrid type inference approach for python," in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 2019-2030.
  18. D. Guo, S. Ren, S. Lu, Z. Feng, D. Tang, S. Liu, L. Zhou, N. Duan, A. Svyatkovskiy, S. Fu et al., "Graphcodebert: Pre-training code repre- sentations with data flow," arXiv preprint arXiv:2009.08366, 2020. · arXiv
  19. M. Allamanis, S. Panthaplackel, and P. Yin, "Unsupervised evaluation of code llms with round-trip correctness," arXiv preprint arXiv:2402.08699, 2024. · arXiv
  20. M. J. Min, Y. Ding, L. Buratti, S. Pujar, G. Kaiser, S. Jana, and B. Ray, "Beyond accuracy: Evaluating self-consistency of code llms," in The Twelfth International Conference on Learning Representations, 2023.
  21. X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y. Chen, J. Feng, C. Sha, X. Peng, and Y. Lou, "Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation," in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024.
  22. J. Henkel, S. K. Lahiri, B. Liblit, and T. Reps, "Code vectors: understanding programs through embedded abstracted symbolic traces," in Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2018. New York, NY, USA: Association for Computing Machinery, 2018, p. 163-174. [Online]. Available: https://doi.org/10.1145/3236024.3236085 · DOI
  23. C. Liu, S. Lu, W. Chen, D. Jiang, A. Svyatkovskiy, S. Fu, N. Sundaresan, and N. Duan, "Code execution with pre-trained language models," arXiv preprint arXiv:2305.05383, 2023. · arXiv
  24. F. Tsimpourlas, G. Rooijackers, A. Rajan, and M. Allamanis, "Embedding and classifying test execution traces using neural networks," IET Software, vol. 16, no. 3, pp. 301-316, 2022. [Online]. Available: https://ietresearch.onlinelibrary.wiley.com/doi/abs/10.1049/sfw2.12038 · DOI
  25. E. Jabbar, H. Hemmati, and R. Feldt, "Investigating execution trace embedding for test case prioritization," in 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS), 2023, pp. 279-290.
  26. E. La Malfa, C. Weinhuber, O. Torre, F. Lin, A. Cohn, N. Shadbolt, and M. Wooldridge, "Code simulation challenges for large language models," arXiv preprint arXiv:2401.09074, 2024. · arXiv
  27. M. Jang, D. S. Kwon, and T. Lukasiewicz, "Becel: Benchmark for consistency evaluation of language models," in Proceedings of the 29th International Conference on Computational Linguistics, 2022, pp. 3680- 3696.
  28. J. Chen, Z. Li, X. Hu, and X. Xia, "Nlperturbator: Studying the robustness of code llms to natural language variations," arXiv preprint arXiv:2406.19783, 2024. · arXiv
  29. M. Jang and T. Lukasiewicz, "Consistency analysis of chatgpt," in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 15 970-15 985.
  30. Y. Elazar, N. Kassner, S. Ravfogel, A. Ravichander, E. Hovy, H. Schütze, and Y. Goldberg, "Measuring and improving consistency in pretrained language models," Transactions of the Association for Computational Linguistics, vol. 9, pp. 1012-1031, 2021.
  31. P. Sahu, M. Cogswell, Y. Gong, and A. Divakaran, "Unpacking large language models with conceptual consistency," arXiv preprint arXiv:2209.15093, 2022. · arXiv
  32. X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery, and D. Zhou, "Self-consistency improves chain of thought reasoning in language models," in The Eleventh International Conference on Learning Representations, 2022.
  33. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288, 2023. · arXiv
  34. A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi et al., "Starcoder 2 and the stack v2: The next generation," arXiv preprint arXiv:2402.19173, 2024. · arXiv
  35. E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou, "Code- gen2: Lessons for training llms on programming and natural languages," arXiv preprint arXiv:2305.02309, 2023. · arXiv
  36. H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, Q. Wang, and T. Xie, "Codereval: A benchmark of pragmatic code generation with generative pre-trained models," in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1-12.
  37. Y. Ding, Z. Wang, W. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth et al., "Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion," Advances in Neural Information Processing Systems, vol. 36, 2024.
  38. Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W.- t. Yih, D. Fried, S. Wang, and T. Yu, "Ds-1000: A natural and reliable benchmark for data science code generation," in International Conference on Machine Learning. PMLR, 2023, pp. 18 319-18 345.
  39. F. Cassano, L. Li, A. Sethi, N. Shinn, A. Brennan-Jones, A. Lozhkov, C. Anderson, and A. Guha, "Can it edit? evaluating the ability of large language models to follow code editing instructions," arXiv preprint arXiv:2312.12450, 2023. · arXiv
  40. M. A. M. Khan, M. S. Bari, X. L. Do, W. Wang, M. R. Parvez, and S. Joty, "xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval," arXiv preprint arXiv:2303.03004, 2023. · arXiv
  41. M. Tufano, S. Chandel, A. Agarwal, N. Sundaresan, and C. Clement, "Predicting code coverage without execution," arXiv preprint arXiv:2307.13383, 2023. · arXiv
  42. H. Hemmati, "How effective are code coverage criteria?" in 2015 IEEE International Conference on Software Quality, Reliability and Security. IEEE, 2015, pp. 151-156.
  43. P. Ammann and J. Offutt, Introduction to software testing. Cambridge University Press, 2016.
  44. B. Souza and M. Pradel, "Lexecutor: Learning-guided execution," in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023, pp. 1522-1534.
  45. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba, "Evaluating large language models trained on code," 2021.
  46. J. Li, G. Li, Y. Zhao, Y. Li, Z. Jin, H. Zhu, H. Liu, K. Liu, L. Wang, Z. Fang et al., "Deveval: Evaluating code generation in practical software projects," arXiv preprint arXiv:2401.06401, 2024. · arXiv
  47. K. Zhang, Z. Li, J. Li, G. Li, and Z. Jin, "Self-edit: Fault-aware code editor for code generation," in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki, Eds. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 769-787. [Online]. Available: https://aclanthology.org/2023.acl-long.45
  48. X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, "Large language models for software engineer- ing: A systematic literature review," arXiv preprint arXiv:2308.10620, 2023. · arXiv
  49. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., "Gpt-4 technical report," arXiv preprint arXiv:2303.08774, 2023. · arXiv
  50. A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier et al., "Mistral 7b," arXiv preprint arXiv:2310.06825, 2023. · arXiv
  51. A. Z. Yang, C. Le Goues, R. Martins, and V. Hellendoorn, "Large language models for test-free fault localization," in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1-12.
  52. A. G. Shypula, A. Madaan, Y. Zeng, U. Alon, J. R. Gardner, Y. Yang, M. Hashemi, G. Neubig, P. Ranganathan, O. Bastani, and A. Yazdanbakhsh, "Learning performance-improving code edits," in The Twelfth International Conference on Learning Representations, 2024. [Online]. Available: https://openreview.net/forum?id=ix7rLVHXyY
  53. Y. Wei, C. S. Xia, and L. Zhang, "Copiloting the copilots: Fusing large language models with completion engines for automated program repair," in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023, pp. 172-184.
  54. J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., "Chain-of-thought prompting elicits reasoning in large language models," Advances in neural information processing systems, vol. 35, pp. 24 824-24 837, 2022.
  55. J. Chen, X. Hu, Z. Li, C. Gao, X. Xia, and D. Lo, "Code search is all you need? improving code suggestions with code search," in Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, 2024, pp. 73:1-73:13.
  56. Y. Song, T. Wang, P. Cai, S. K. Mondal, and J. P. Sahoo, "A compre- hensive survey of few-shot learning: Evolution, applications, challenges, and opportunities," ACM Computing Surveys, vol. 55, no. 13s, pp. 1-40, 2023.
  57. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, "Efficient memory management for large language model serving with pagedattention," in Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
  58. S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan, "Tree of thoughts: Deliberate problem solving with large language models," Advances in Neural Information Processing Systems, vol. 36, 2024.

Cited by

Metadata

Authors
Chen, J., Pan, Z., Hu, X., Li, Z., Li, G., & Xia, X.
Venue
arXiv
Publisher
arXiv
Published
2024
DOI
10.48550/arxiv.2403.16437
arXiv
2403.16437
References
0
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.