CoChat
Sign Up
PreprintCited by 70Software Testing and Debugging TechniquesSoftware Engineering ResearchTeaching and Learning Programming

Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step

Open access
Zhong, Li, Wang, Zilong, Shang, Jingbo·2024·arXiv
DOI 10.48550/arxiv.2402.16906arXiv 2402.16906
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) are leading significant progress in code generation. Beyond one-pass code generation, recent works further integrate unit tests and program verifiers into LLMs to iteratively refine the generated programs. However, these works consider the generated programs as an indivisible entity, which falls short for LLMs in debugging the programs, especially when the programs contain complex logic flows and data operations. In contrast, when human developers debug programs, they typically set breakpoints and selectively examine runtime execution information. The execution flow and the intermediate variables play a crucial role in the debugging process, yet they are underutilized in the existing literature on code generation. In this study, we introduce Large Language Model Debugger (LDB), a novel debugging framework that enables LLMs to refine their generated programs with the runtime execution information. Specifically, LDB segments the programs into basic blocks and tracks the values of intermediate variables after each block throughout the runtime execution. This allows LLMs to concentrate on simpler code units within the overall execution flow, verify their correctness against the task description block by block, and efficiently pinpoint any potential errors. Experiments demonstrate that LDB consistently enhances the baseline performance by up to 9.8% across the HumanEval, MBPP, and TransCoder benchmarks, archiving new state-of-the-art performance in code debugging for various LLM selections.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.16906

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A prevalent paradigm prompts frozen models to iteratively refine their outputs using external feedback.

    “A prevalent paradigm prompts frozen models to iteratively refine their outputs using external feedback.”

    From ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning (2026) · cites this paper as Zhong et al., 2024

    The “Feedback-based Code Refinement” section describes an iterative refinement paradigm that uses feedback, including interpreter outputs, and says LDB uses the original large language model rather than training an additional bug-fixing model. The “J Prompt Example in LDB” section shows a model prompted to generate a program and then regenerate it with debugging feedback. The paper addresses code outputs, however, and does not establish that frozen models are a prevalent paradigm for refining outputs generally. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Iterative refinement methods depend on high-quality external oracles (e.g., compilers, test suites, or critic models).

    “Despite their promise, these methods depend on high-quality external oracles (e.g., compilers, test suites, or critic models), which may be unavailable or expensive in real-world deployment.”

    From ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning (2026) · cites this paper as Zhong et al., 2024

    The Feedback-based Code Refinement section describes iterative methods using interpreter output and other feedback, and the Limitation section says correct test cases are mandatory for LDB. The paper does not establish that iterative refinement methods generally depend on high-quality external oracles such as compilers or critic models, or that those oracles may be unavailable or expensive in deployment. Our copy provides no page numbers for these sections. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • LLMs have achieved strong performance in generating source code from natural language descriptions.

    “In parallel, LLMs have achieved strong performance in generating source code from natural language descriptions [16, 22, 48, 76, 77].”

    From A Multi-Agent Framework for Accessible Web Code Generation (2026) · cites this paper as [77]

  • In the original HumanEval benchmark, test cases were provided for only 161 of 164 tasks.

    “This version includes internal test cases for all 164 tasks (in the original benchmark, test cases were provided for only 161 of 164 tasks, as detailed in Section 6).”

    From AI Agents That Matter (2024) · cites this paper as [64]

    Section 3.1 (page unavailable) says the authors extracted visible HumanEval test cases from task descriptions, but it gives no count of tasks with test cases. Searches for “161,” “164,” and missing test cases found no passage establishing that test cases were provided for only 161 of 164 tasks in the original benchmark. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 71

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. · arXiv
  2. Frances E Allen. 1970. Control flow analysis. ACM Sigplan Notices, 5(7):1-19.
  3. Glenn Ammons and James R Larus. 1998. Improving data-flow analysis with path profiles. In Proceedings of the ACM SIGPLAN 1998 conference on Program- ming language design and implementation, pages 72-84.
  4. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. · arXiv
  5. Thomas Ball and James R Larus. 1994. Optimally pro- filing and tracing programs. ACM Transactions on Programming Languages and Systems (TOPLAS), 16(4):1319-1360.
  6. Thomas Ball and James R Larus. 1996. Efficient path profiling. In Proceedings of the 29th Annual IEEE/ACM International Symposium on Microarchi- tecture. MICRO 29, pages 46-57. IEEE.
  7. Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R Bow- man, Kyunghyun Cho, and Ethan Perez. 2023a. Im- proving code generation by training with natural lan- guage feedback. arXiv preprint arXiv:2303.16749. · arXiv
  8. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. Codet: Code generation with generated tests. In The Eleventh International Conference on Learning Representations.
  9. Hongqiao Chen, Kexun Zhang, Lei Li, and William Yang Wang. 2023b. Tooldec: Syntax error-free and generalizable tool use for llms via finite-state decoding. In The 3rd Workshop on Mathematical Reasoning and AI at NeurIPS'23.
  10. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. · arXiv
  11. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023c. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128. · arXiv
  12. Xinyun Chen, Chang Liu, and Dawn Song. 2018. Execution-guided neural program synthesis. In Inter- national Conference on Learning Representations.
  13. Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, et al. 2022. Binding language models in symbolic languages. In The Eleventh International Conference on Learning Representations.
  14. John Cocke. 1970. Global common subexpression elim- ination. In Proceedings of a symposium on Compiler optimization, pages 20-24.
  15. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Ja- son Weston. 2023. Chain-of-verification reduces hal- lucination in large language models. arXiv preprint arXiv:2309.11495. · arXiv
  16. Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roy- choudhury, and Shin Hwei Tan. 2023. Automated repair of programs from large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1469-1481. IEEE.
  17. Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. 2023. In-context autoencoder for context com- pression in a large language model. arXiv preprint arXiv:2307.06945. · arXiv
  18. Qiuhan Gu. 2023. Llm-based code generation method for golang compiler testing. In Proceedings of the 31st ACM Joint European Software Engineering Con- ference and Symposium on the Foundations of Soft- ware Engineering, pages 2201-2203.
  19. Priyanshu Gupta, Avishree Khare, Yasharth Bajpai, Saikat Chakraborty, Sumit Gulwani, Aditya Kanade, Arjun Radhakrishna, Gustavo Soares, and Ashish Ti- wari. 2023. Grace: Language models meet code edits. In Proceedings of the 31st ACM Joint European Soft- ware Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1483- 1495.
  20. Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. 2023. Metagpt: Meta programming for multi-agent collabo- rative framework. arXiv preprint arXiv:2308.00352. · arXiv
  21. Mengkang Hu, Yao Mu, Xinmiao Yu, Mingyu Ding, Shiguang Wu, Wenqi Shao, Qiguang Chen, Bin Wang, Yu Qiao, and Ping Luo. 2023. Tree-planner: Efficient close-loop task planning with large language models. arXiv preprint arXiv:2310.08582. · arXiv
  22. Xueyu Hu, Kun Kuang, Jiankai Sun, Hongxia Yang, and Fei Wu. 2024. Leveraging print debugging to improve code generation in large language models. arXiv preprint arXiv:2401.05319. · arXiv
  23. Baizhou Huang, Shuai Lu, Weizhu Chen, Xiaojun Wan, and Nan Duan. 2023a. Enhancing large language models in coding through multi-perspective self- consistency. arXiv preprint arXiv:2309.17272. · arXiv
  24. Haochen Huang, Bingyu Shen, Li Zhong, and Yuanyuan Zhou. 2023b. Protecting data integrity of web ap- plications with database constraints inferred from application code. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pages 632-645.
  25. Haochen Huang, Chengcheng Xiang, Li Zhong, and Yuanyuan Zhou. 2021. {PYLIVE}:{On-the-Fly} code change for python-based online services. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 349-363.
  26. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xiny- ing Song, and Denny Zhou. 2023c. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798. · arXiv
  27. Kai Huang, Xiangxin Meng, Jian Zhang, Yang Liu, Wenjie Wang, Shuhao Li, and Yuqing Zhang. 2023d. An empirical study on fine-tuning large language models of code for automated program repair. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1162- 1174. IEEE.
  28. Shuyang Jiang, Yuhao Wang, and Yu Wang. 2023. Self- evolve: A code evolution framework via large lan- guage models. arXiv preprint arXiv:2306.02907. · arXiv
  29. Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svy- atkovskiy. 2023. Inferfix: End-to-end program repair with llms. arXiv preprint arXiv:2303.07263. · arXiv
  30. James R Larus. 1999. Whole program paths. ACM SIGPLAN Notices, 34(5):259-269.
  31. Hung Le, Hailin Chen, Amrita Saha, Akash Gokul, Doyen Sahoo, and Shafiq Joty. 2023. Codechain: To- wards modular code generation through chain of self- revisions with representative sub-modules. arXiv preprint arXiv:2310.08992. · arXiv
  32. Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314-21328.
  33. Jian Li, Yue Wang, Michael R. Lyu, and Irwin King. 2018. Code completion with neural attention and pointer networks. In Proceedings of the 27th Inter- national Joint Conference on Artificial Intelligence, IJCAI'18, page 4159-25. AAAI Press.
  34. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161. · arXiv
  35. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092-1097.
  36. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. arXiv preprint arXiv:2305.20050. · arXiv
  37. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2023. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651. · arXiv
  38. Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. 2023. Octopack: Instruction tun- ing code large language models. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Fol- lowing.
  39. Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Alex Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao. 2022. Learning math reasoning from self-sampled correct and partially-correct solutions. In The Eleventh International Conference on Learn- ing Representations.
  40. Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoy- anov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. In International Con- ference on Machine Learning, pages 26106-26128.
  41. PMLR.
  42. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations.
  43. Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334. · arXiv
  44. Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2023. Ex- amining zero-shot vulnerability repair with large lan- guage models. In 2023 IEEE Symposium on Security and Privacy (SP), pages 2339-2356. IEEE.
  45. Reese T Prosser. 1959. Applications of boolean ma- trices to the analysis of flow diagrams. In Papers presented at the December 1-3, 1959, eastern joint IRE-AIEE-ACM computer conference, pages 133- 138.
  46. Zeeshan Rasheed, Muhammad Waseem, Mika Saari, Kari Systä, and Pekka Abrahamsson. 2024. Code- pori: Large scale model for autonomous software development by using multi-agents. arXiv preprint arXiv:2402.01411. · arXiv
  47. Veselin Raychev, Martin Vechev, and Eran Yahav. 2014. Code completion with statistical language models. In Proceedings of the 35th ACM SIGPLAN conference on programming language design and implementa- tion, pages 419-428.
  48. Eric Rotenberg, Steve Bennett, and James E Smith. 1996. Trace cache: a low latency approach to high bandwidth instruction fetching. In Proceedings of the 29th Annual IEEE/ACM International Symposium on Microarchitecture. MICRO 29, pages 24-34. IEEE.
  49. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950. · arXiv
  50. Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanus- sot, and Guillaume Lample. 2020. Unsupervised translation of programming languages. Advances in Neural Information Processing Systems, 33:20601- 20611.
  51. Timothy Sherwood, Erez Perelman, and Brad Calder. 2001. Basic block distribution analysis to find peri- odic behavior and simulation points in applications. In Proceedings 2001 International Conference on Parallel Architectures and Compilation Techniques, pages 3-14. IEEE.
  52. Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I Wang. 2022. Natural lan- guage to code translation with execution. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3533-3546.
  53. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. 2023. Re- flexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems.
  54. Zilu Tang, Mayank Agarwal, Alexander Shypula, Bailin Wang, Derry Wijaya, Jie Chen, and Yoon Kim. 2023. Explain-then-translate: an analysis on improving pro- gram translation with self-generated explanations. In Findings of the Association for Computational Lin- guistics: EMNLP 2023, pages 1741-1788.
  55. Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Mar- tin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, et al. 2024. Chain-of-table: Evolving tables in the reasoning chain for table understanding. arXiv preprint arXiv:2401.04398. · arXiv
  56. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837.
  57. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023. Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01693. · arXiv
  58. Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 440-450.
  59. Zhiqiang Yuan, Junwei Liu, Qiancheng Zi, Ming- wei Liu, Xin Peng, and Yiling Lou. 2023. Eval- uating instruction-tuned large language models on code comprehension and generation. arXiv preprint arXiv:2308.01240. · arXiv
  60. Jialu Zhang, José Cambronero, Sumit Gulwani, Vu Le, Ruzica Piskac, Gustavo Soares, and Gust Ver- bruggen. 2022a. Repairing bugs in python assign- ments using large language models. arXiv preprint arXiv:2209.14876. · arXiv
  61. Jialu Zhang, De Li, John Charles Kolesar, Hanyuan Shi, and Ruzica Piskac. 2022b. Automated feedback gen- eration for competition-level code. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1-13.
  62. Jialu Zhang, Todd Mytkowicz, Mike Kaufman, Ruz- ica Piskac, and Shuvendu K Lahiri. 2022c. Using pre-trained language models to resolve textual and semantic merge conflicts (experience paper). In Pro- ceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 77-88.
  63. Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. 2023a. Algo: Synthesizing algorithmic programs with generated oracle verifiers. arXiv preprint arXiv:2305.14591. · arXiv
  64. Tianyi Zhang, Tao Yu, Tatsunori Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida Wang. 2023b. Coder reviewer reranking for code generation. In International Conference on Machine Learning, pages 41832-41846.
  65. PMLR.
  66. Li Zhong. 2023. A survey of prevent and de- tect access control vulnerabilities. arXiv preprint arXiv:2304.10600. · arXiv
  67. Li Zhong and Zilong Wang. 2023. A study on robust- ness and reliability of large language model code generation. arXiv preprint arXiv:2308.10335. · arXiv
  68. Li Zhong, Chengcheng Xiang, Haochen Huang, Bingyu Shen, Eric Mugnier, and Yuanyuan Zhou. 2024. Ef- fective bug detection with unused definitions. In Proceedings of the Nineteenth European Conference on Computer Systems, pages 720-735.
  69. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2023a. Lan- guage agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406. · arXiv
  70. Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. 2023b. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921. · arXiv
  71. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, et al. 2022. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh Interna- tional Conference on Learning Representations.

Cited by

Sorted by citations

Related

Authors
Zhong, L., Wang, Z., & Shang, J.
Affiliations
University of California San Diego (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
851-870
DOI
10.48550/arxiv.2402.16906
arXiv
2402.16906
OpenAlex
W4402670687
Semantic Scholar
c6c584ea3983627329eebff7c78951938b227351
Cited by
69
References
0
Citations by year
2026: 27 · 2025: 36 · 2024: 6
FWCI
18.84
Type
conference paper
Language
en
Open access
Licence
Locations
  • Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (2333790, CAREER, 2239440)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
debugging, basic blocks, code generation, program verification, HumanEval, MBPP, transcoder

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review