The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
Open accessAbstract
Program synthesis has been long studied with recent approaches focused on directly using the power of Large Language Models (LLMs) to generate code. Programming benchmarks, with curated synthesis problems and test-cases, are used to measure the performance of various LLMs on code synthesis. However, these test-cases can be limited in both quantity and quality for fully assessing the functional correctness of the generated code. Such limitation in the existing benchmarks begs the following question: In the era of LLMs, is the code generated really correct? To answer this, we propose EvalPlus -- a code synthesis evaluation framework to rigorously benchmark the functional correctness of LLM-synthesized code. EvalPlus augments a given evaluation dataset with large amounts of test-cases newly produced by an automatic test input generator, powered by both LLM- and mutation-based strategies. While EvalPlus is general, we extend the test-cases of the popular HumanEval benchmark by 80x to build HumanEval+. Our extensive evaluation across 26 popular LLMs (e.g., GPT-4 and ChatGPT) demonstrates that HumanEval+ is able to catch significant amounts of previously undetected wrong code synthesized by LLMs, reducing the pass@k by up-to 19.3-28.9%. We also surprisingly found that test insufficiency can lead to mis-ranking. For example, both WizardCoder-CodeLlama and Phind-CodeLlama now outperform ChatGPT on HumanEval+, while none of them could on HumanEval. Our work not only indicates that prior popular code synthesis evaluation results do not accurately reflect the true performance of LLMs for code synthesis, but also opens up a new direction to improve such programming benchmarks through automated testing. We have open-sourced our tools, enhanced datasets as well as all LLM-generated code at https://github.com/evalplus/evalplus to facilitate and accelerate future LLM-for-code research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Similarly, MBPP+ offers 35 times more test cases than the original MBPP.”
From Qwen2.5-Coder Technical Report (2024) · cites this paper as Liu et al., 2023
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For the second case, a finite test suite necessarily under-approximates an executable's full specification (Liu et al., 2023; Le Goues et al., 2015).”
From ProgramBench: Can Language Models Rebuild Programs From Scratch? (2026) · cites this paper as Liu et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As a result, LLMs frequently produce code that fails to compile, violates existing patterns, or ignores critical constraints when the relevant context exceeds their window size [22].”
From LongCodeZip: Compress Long Context for Code Language Models (2025) · cites this paper as [22]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“For example, some subsample instances from existing benchmarks that cannot be solved by current models [93, 82], or augment them to construct more challenging evaluations [21, 44, 49].”
From SciCode: A Research Coding Benchmark Curated by Scientists (2024) · cites this paper as [49]
The cited source says what the claim attributes to it.
In the citing paper
“To further ensure accurate evaluation, EvalPlus (Liu et al., 2023) extends HumanEval into HumanEval+ by adding 80 times more unique test cases and correcting inaccurate groundtruth solutions in HumanEval.”
From Qwen2.5-Coder Technical Report (2024) · cites this paper as Liu et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“While passing these tests provides some evidence of functional correctness[25], we deliberately exclude them from our core success metric-since test cases offer only partial coverage and cannot guarantee semantic soundness (see Section 2 for discussion).”
From CLEVER: A Curated Benchmark for Formally Verified Code Generation (2025) · cites this paper as [25]
The cited source says what the claim attributes to it.
In the citing paper
“While passing these tests provides some evidence of functional correctness[25], we deliberately exclude them from our core success metric-since test cases offer only partial coverage and cannot guarantee semantic soundness (see Section 2 for discussion).”
From CLEVER: A Curated Benchmark for Formally Verified Code Generation (2025) · cites this paper as [25]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“When a benchmark's accuracy saturates, it is often retired and replaced with a more challenging version.”
From Life After Benchmark Saturation: A Case Study of CORE-Bench (2026) · cites this paper as [35]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 61
- T. Ahmed and P. Devanbu. Few-shot training llms for project-specific code-summarization. In 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1-5, 2022.
- L. B. Allal, R. Li, D. Kocetkov, C. Mou, C. Akiki, C. M. Ferrandis, N. Muennighoff, M. Mishra, A. Gu, M. Dey, et al. Santacoder: don't reach for the stars! arXiv preprint arXiv:2301.03988, 2023. · arXiv
- S. Bang, S. Nam, I. Chun, H. Y. Jhoo, and J. Lee. Smt-based translation validation for machine learning compiler. In Computer Aided Verification: 34th International Conference, CAV 2022, Haifa, Israel, August 7-10, 2022, Proceedings, Part II, pages 386-407. Springer, 2022.
- T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
- C. Cadar, D. Dunbar, D. R. Engler, et al. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, volume 8, pages 209-224, 2008.
- F. Cassano, J. Gouwar, D. Nguyen, S. Nguyen, L. Phipps-Costin, D. Pinckney, M.-H. Yee, Y. Zi, C. J. Anderson, M. Q. Feldman, et al. Multipl-e: A scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, 2023.
- S. K. Cha, M. Woo, and D. Brumley. Program-adaptive mutational fuzzing. In 2015 IEEE Symposium on Security and Privacy, pages 725-741. IEEE, 2015.
- M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. · arXiv
- Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In 32nd International Symposium on Software Testing and Analysis (ISSTA), 2023.
- Y. Deng, C. S. Xia, C. Yang, S. D. Zhang, S. Yang, and L. Zhang. Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt. In 46th International Conference on Software Engineering (ICSE), 2024.
- U. Feige. A threshold of ln n for approximating set cover. Journal of the ACM (JACM), 45(4):634-652, 1998.
- D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. Yih, L. Zettlemoyer, and M. Lewis. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2023.
- C. Green. Application of theorem proving to problem solving. In Readings in Artificial Intelligence, pages 202-222. Elsevier, 1981.
- S. Gulwani. Automating string processing in spreadsheets using input-output examples. SIGPLAN Not., 46(1):317-330, jan 2011.
- S. Gulwani, O. Polozov, and R. Singh. Program synthesis. Foundations and Trends® in Programming Languages, 4(1-2):1-119, 2017.
- T. Hastie, R. Tibshirani, J. H. Friedman, and J. H. Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
- C. Holler, K. Herzig, and A. Zeller. Fuzzing with code fragments. In 21st USENIX Security Sym- posium (USENIX Security 12), pages 445-458, Bellevue, WA, Aug. 2012. USENIX Association.
- M. Ivanković, G. Petrović, R. Just, and G. Fraser. Code coverage at google. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 955-963, 2019.
- S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1643-1652, 2018.
- N. Jiang, K. Liu, T. Lutellier, and L. Tan. Impact of code language models on automated program repair. arXiv preprint arXiv:2302.05020, 2023. · arXiv
- C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770, 2023. · arXiv
- A. Kalyan, A. Mohta, O. Polozov, D. Batra, P. Jain, and S. Gulwani. Neural-guided deductive search for real-time program synthesis from examples. In International Conference on Learning Representations, 2018.
- J. C. King. Symbolic execution and program testing. Communications of the ACM, 19(7):385-394, 1976.
- M.-A. Lachaux, B. Roziere, L. Chanussot, and G. Lample. Unsupervised translation of programming languages. arXiv preprint arXiv:2006.03511, 2020. · arXiv
- K. R. M. Leino. Dafny: An automatic program verifier for functional correctness. In International conference on logic for programming artificial intelligence and reasoning, pages 348-370. Springer, 2010.
- Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092-1097, 2022.
- J. Liu, Y. Wei, S. Yang, Y. Deng, and L. Zhang. Coverage-guided tensor compiler fuzzing with joint ir-pass mutation. Proceedings of the ACM on Programming Languages, 6(OOPSLA1):1-26, Apr. 2022.
- Z. Liu, C. Chen, J. Wang, X. Che, Y. Huang, J. Hu, and Q. Wang. Fill in the blank: Context-aware automated text input generation for mobile gui testing. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 1355-1367. IEEE, 2023.
- N. P. Lopes, J. Lee, C.-K. Hur, Z. Liu, and J. Regehr. Alive2: bounded translation validation for llvm. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation, pages 65-79, 2021.
- S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664, 2021. · arXiv
- Z. Luo, C. Xu, P. Zhao, Q. Sun, X. Geng, W. Hu, C. Tao, J. Ma, Q. Lin, and D. Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568, 2023. · arXiv
- Z. Manna and R. J. Waldinger. Toward automatic program synthesis. Communications of the ACM, 14(3):151-165, 1971.
- W. M. McKeeman. Differential testing for software. Digital Technical Journal, 10(1):100-107, 1998.
- B. Meyer. Applying'design by contract'. Computer, 25(10):40-51, 1992.
- B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of unix utilities. Communications of the ACM, 33(12):32-44, 1990.
- G. C. Necula. Translation validation for an optimizing compiler. In Proceedings of the ACM SIGPLAN 2000 conference on Programming language design and implementation, pages 83-94, 2000.
- E. Nijkamp, H. Hayashi, C. Xiong, S. Savarese, and Y. Zhou. Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint, 2023.
- E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2023.
- P. Oehlert. Violating assumptions with fuzzing. IEEE Security & Privacy, 3(2):58-62, 2005.
- OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. · arXiv
- K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
- G. Petrović and M. Ivanković. State of mutation testing at google. In Proceedings of the 40th international conference on software engineering: Software engineering in practice, pages 163-171, 2018.
- G. Rothermel, M. J. Harrold, J. Von Ronne, and C. Hong. Empirical studies of test-suite reduction. Software Testing, Verification and Reliability, 12(4):219-249, 2002.
- B. Rozière, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, T. Remez, J. Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. · arXiv
- B. Roziere, J. M. Zhang, F. Charton, M. Harman, G. Synnaeve, and G. Lample. Leveraging automated unit tests for unsupervised code translation. arXiv preprint arXiv:2110.06773, 2021. · arXiv
- K. Serebryany. Continuous fuzzing with libfuzzer and addresssanitizer. In 2016 IEEE Cybersecurity Development (SecDev), pages 157-157. IEEE, 2016.
- D. E. Shaw, W. R. Swartout, and C. C. Green. Inferring lisp programs from examples. In IJCAI, volume 75, pages 260-267, 1975.
- A. Shi, A. Gyori, M. Gligoric, A. Zaytsev, and D. Marinov. Balancing trade-offs in test-suite reduction. In Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, pages 246-256, 2014.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
- R. J. Waldinger and R. C. Lee. Prow: A step toward automatic program writing. In Proceedings of the 1st international joint conference on Artificial intelligence, pages 241-252, 1969.
- Y. Wang, H. Le, A. D. Gotmare, N. D. Bui, J. Li, and S. C. Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922, 2023. · arXiv
- Y. Wei, C. S. Xia, and L. Zhang. Copiloting the copilots: Fusing large language models with completion engines for automated program repair. arXiv preprint arXiv:2309.00608, 2023. · arXiv
- D. Winterer, C. Zhang, and Z. Su. On the unusual effectiveness of type-aware operator mutations for testing smt solvers. Proceedings of the ACM on Programming Languages, 4(OOPSLA):1-25, 2020.
- C. S. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L. Zhang. Universal fuzzing via large language models. In 46th International Conference on Software Engineering (ICSE), 2024.
- C. S. Xia, Y. Wei, and L. Zhang. Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery, 2023.
- C. S. Xia and L. Zhang. Less training, more repairing please: revisiting automated program re- pair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 959-971, 2022.
- F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1-10, 2022.
- X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and understanding bugs in c compilers. In Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '11, page 283-294, New York, NY, USA, 2011. Association for Computing Machinery.
- T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
- L. Zhang, D. Marinov, L. Zhang, and S. Khurshid. An empirical study of junit test-suite reduction. In 2011 IEEE 22nd International Symposium on Software Reliability Engineering, pages 170-179. IEEE, 2011.
- Q. Zheng, X. Xia, X. Zou, Y. Dong, S. Wang, Y. Xue, Z. Wang, L. Shen, A. Wang, Y. Li, et al. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568, 2023. · arXiv
Cited by
Sorted by citations
Items 301–400 of 426
Testing Android Third Party Libraries with LLMs to Detect Incompatible APIs
Mahmud, T., Duan, B., Che, M., Ngu, A. H. H., & Yang, G. (2025) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1109/forge66646.2025.00039
Zeng, Z., Ratnavelu, K., & Huat, O. S. (2025) · WSEAS TRANSACTIONS ON COMPUTER RESEARCH · doi:10.37394/232018.2025.13.42
Haase, J. (2025) · Organisationsberatung, Supervision, Coaching · doi:10.1007/s11613-025-00946-x
Yan, L., Lyu, C., Xing, R., Li, W., Samih, Y., Ji, T., Wang, L., & Jiang, S. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-9994-0_12
Zhou, Y., Shan, Z., Dai, Z., & Zhang, Z. (2025) · International Conference on Artificial Intelligence and Educational Systems · doi:10.1145/3744367.3744402
“(1) Correctness Metric: Adapting the EvalPlus framework [ 6 ] originally developed for strict evaluation of code functionality, this study employs a standardized scoring method based on the test case passing rate: Here, N passed denotes the number of test cases passed, and N total represents the…”
Automated configuration for cost-effective digital solutions
Ling, Z., McFarlane, D., Brooks, S., de Silva, L., Hawkridge, G., & Thorne, A. (2025) · Computers in Industry · doi:10.1016/j.compind.2025.104397
Environment-Aware Code Generation: How far are We?
Wu, T., Chen, R., Du, W., Ma, S., Qi, G., Xing, Z., Khadivi, S., Periyathambi, R., & Haffari, G. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787784 · Cited by 5
“Although LLMs demonstrate strong capabilities on standard code generation benchmarks [5, 6, 9, 13, 18, 26, 28, 30, 49], their outputs frequently encounter failures when deployed in actual development environments [24, 48].”
FLAME: Enhancing Functional Coverage in Processor Verification via Large Language Models
Li, X., Chen, J., Yan, M. X., Wang, D., Fan, X., Tang, Z., Kai, S., Hao, J., Yuan, M., & Wang, Z. (2026) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2026.3666802
“Note that FLAME chooses the source code level rather than the instruction level in order to improve test validity, since LLMs have demonstrated superior performance on source code generation [17], [18], [19].”
SpiceFuzz: LLM-Based Fuzzing for Spice Circuit Simulator Tools Bug Detection
Ren, Z., Liu, H., Guo, S., Guo, Y., Li, X., & Jiang, H. (2026) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3797154
“Recently, LLMs have shown remarkable performance in many tasks, such as code generation [26], text summarization [27], and classification [28].”
Vu, T. T., Bui, T.-D., Nguyen, T.-T., Nguyen, S., & Vo, H. D. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112852 · Cited by 3
Tang, K., Tong-Ho, H. N. H., Riegel, K., Denny, P. C., Giacaman, N., & Leinonen, J. (2026) · Australasian Computing Education Conference · doi:10.1145/3786228.3786249
“This underlying model has been shown to perform well on rigorous code-generation benchmarks [21], and is thus suitable for powering our agent.”
Toward a Lightweight Mixture of Experts with Future-Aware Auxiliary Tuning
Hoàng, N. H., Linh, N. T. T., Quyet, H. M., & Ha, N. V. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-21628-1_13
Poimanov, D. R., & Shutov, M. S. (2025) · Automatic Documentation and Mathematical Linguistics · doi:10.3103/s0005105525701389
Evaluating LLMs Code Reasoning Under Real-World Context
Liu, C. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3787747
Minkova, L., Bouhandi, M., Alassan, M. S. Y., Dahhane, W., & Ettifouri, E. H. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-18716-1_10
Li, M., & Krishnamachari, B. (2026) · IEEE SoutheastCon · doi:10.1109/southeastcon63549.2026.11476431
TDPGen: Optimizing Agentic Code Generation via Test-Driven Planning and Hierarchical ReAct
Li, M., Yuan, S., & Han, M. (2026) · International Symposium on Computer Applications and Information Systems (ISCAIS) · doi:10.1109/iscait69154.2026.11477183
“To empirically validate the efficacy of the TDPGen framework, we conduct extensive experiments across several mainstream benchmarks within the code synthesis domain: HumanEval [12], MBPP [13], and their enhanced variants, HumanEval-ET [14] and MBPP-ET [14].”
Lin, Z., Shen, Y., Cai, Q., Sun, H., Zhou, J., & Xiao, M. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462205 · Cited by 3
Audio-Text Jailbreak Attack on Large Audio-Language Models: Towards Generality and Stealthiness
Li, Y., Zhang, J., Chen, Y., Zhao, Z., & Wang, J. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464814
Self-Chill: A Divergence-Convergence Framework for Multi-Path Generation in LLMs
Li, Y., Chen, Y., Li, D., Wang, W. B., & Zhang, Y. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464445
IM-RACG: Information Density-Based Adaptive Masking Strategy for Retrieval-Augmented Code Generation
Shi, C., Gao, M., Li, B., Fan, Y., & Gao, Z. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11462224
AICodeDetect: A Pipeline for Systematic Detection and Analysis of AI-Generated Code
Vijayakumar, S., Negrin, P., & Faloutsos, C. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-20362-5_10
A Unified Framework for Evaluating Mathematical Problem Solving Ability of Large Language Models
Quan, L., & Sun, C. (2025) · International Conference on Intelligent Education and Intelligent Research (IEIR) · doi:10.1109/ieir67391.2025.11491027
Benchmarking AI Models for Automated Code Generation and Testing Using HumanEval
Manzoor, M., Rehman, M. U., Kazim, M., Deka, L., Khattak, A. A., & Usmani, I. A. (2026) · Global Conference on Wireless and Optical Technologies (GCWOT) · doi:10.1109/gcwot69191.2026.11499481
Entropy-Aware Watermarking for Code Generation Models
Shi, Y., Bao, S., Wu, H., Ye, J., & Zhang, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-7820-7_13
Xie, X., Chen, Y., Luo, L., & Cui, Y. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-7731-6_30
A multi-dimensional test case evaluation framework based on clustering and differential testing
Jiang, D., Zhou, J., Wang, X., Fan, X., Liu, H., & Liu, Y. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112936
Wily: High-Performance Complexity Gated-Feedback for AI Coding Agents
Shaw, A., & Beheshti, A. (2026) · ACM Conference on AI and Agentic Systems (CAIS) · doi:10.1145/3786335.3813220
“Liu et al. [11] later showed that pass rates drop 19–29% when evaluated with 80 × more rigorous tests, underscoring the gap between functional correctness and true quality.”
Quality Assurance of Large Language Model-based Software: A Systematic Literature ReviewPreprint
O'Berry, J., & Kanewala, U. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6864773
Large Language Models for Automated Software Test Generation
George, J. (2026) · International Journal of Information Technology Research Studies (IJITRS) · doi:10.63090/ijitrs/3139.3209.0022
ChatTest: Coverage-Enhanced Testbench Generation for Agile Hardware Verification with LLMs
Wan, G.-W., Su, S., Zhang, J., Wong, S., Xing, M., Ji, L., Jiang, Z., Wang, X. W., & Yang, J. (2026) · Design, Automation, and Test in Europe · doi:10.23919/date69613.2026.11539733 · Cited by 1
“Building upon their demonstrated success in software engineering [11], LLMs have spurred growing interest in LLM-Aided Design (LAD) [12].”
SkillCraft: LLM Powered Interview Preparation Platform
Shabari Shedthi, B., Acharya, A., Anvesh, D., & Reeja, S. R. (2026) · International Conference on Artificial Intelligence and Data Engineering (AIDE) · doi:10.1109/aide69088.2026.11544404
“The paper [6] gives the limitations of present code evaluation datasets for evaluating the practical perfection of code produced by LLMs. Introduced EvalPlus, a benchmarking framework to evaluate the exactness of LLM synthesized code by generating and diversifying huge quantities of new test…”
Zhang, Y., Yao, K., Zhang, L., & Zhao, C. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133242 · Cited by 1
Self-citationInoue, S., Morisaki, S., Wada, T., & Hattori, Y. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3702933
“Leveraging generative AI for software development [14], [15], [16], [17], [18], [19], [20] is an active research area.”
Giagnorio, A., Martin-Lopez, A., & Bavota, G. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3703553
“In recent years, numerous code generation techniques [1], [3], [46], [47] and benchmarks [11], [12], [48], [49] have been proposed, with most of them targeting popular programming languages, such as Python and Java.”
Kim, M., Choi, M., Wang, E., & Chung, S. (2026) · The Journal of Korean Institute of Information Technology · doi:10.14801/jkiit.2026.24.5.75
Spec2FaaS: using AI Agents for Autonomous Serverless Function Development and Deployment
Lupini, M., Russo, G. R., & Cardellini, V. (2026) · Distributed Event-Based Systems · doi:10.1145/3809481.3812613
Sivapiran, S., & Uddin, G. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808123 · Cited by 1
“This 118-instance benchmark [29] measures the percentage of solutions meeting reference efficiency.”
Compiling Code LLMs into Lightweight Executables
Shi, J., He, J., Yang, Z., Yang, C., Klymenko, M. V., Hoang, T., Xu, X., Xing, Z., & Lo, D. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808196
Liu, K., Pan, Y., Du, Y., Zhang, L., He, D., & Xiang, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3817119
Self-citationThe transformative potential of AI in software engineering: a case study on LeetCode and ChatGPT
Merkel, M., & Dörpinghaus, J. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10912-5 · Cited by 7
“In contrast, the authors from Liu et al. (2023) employed the HumanEval dataset, expanded it, and conducted an analysis with 26 different GenAIs on 12 Microsoft Copilot, https://www .microsoft. com/en-us/m icrosoft -copilot/learn?ep=0&form=MA13LV&es=31 13 Hugging Face StarCoder,…”
PromptCOS: Towards Content-Only System Prompt Copyright Auditing for LLMs
Yang, Y., Li, Y., Yao, H., Huang, E., Shao, S., Wang, Y., Wang, Z., Tao, D., & Qin, Z. (2026) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp63933.2026.00142
“For coding, we employ Hu-manEval [27], a canonical dataset released by OpenAI that contains 164 high-quality programming tasks.”
Self-citationCan LLMs Keep Up with Library Changes? An Exploratory Study on LLM-Generated Code
Lin, X., Liu, J., & Bao, L. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00088
Self-citationAn Empirical Investigation on the Use of Large Language Models for Performance Bug Detection
Imran, M., Cortellessa, V., Ruscio, D. D., Rubei, R., & Traini, L. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00044
“These models were selected among instruction-tuned, code-pretrained LLMs that matched our criteria, and were also included in recent leaderboards such as Chatbot Arena [39] and the evaluation by Liu et al. [40].”
Programming Language Confusion: When Code LLMs Can't Keep Their Languages Straight
Moumoula, M. B., Nikiema, S. L., Kaboré, A. K., Klein, J., & Bissyandé, T. F. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00087 · Cited by 6
“Studies have demonstrated their ability to automate repetitive coding tasks [10], [11], [12], [13], potentially reducing implementation time [14], augmenting programmer efficacy [15], and expanding access to complex programming techniques for developers with varied skill levels [16].”
Leveraging Enhanced Test-Driven Development for Accurate Code Generation in LLMs
Zhang, R., Shan, W., Fu, C., & Long, T. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00089
“The optimized code not only passed the original test suite but also validated against the extended edge cases from EvalPlus [27].”
On the Time Efficiency of Code Generated and Optimized by Large Language Models
Peng, Y., Wan, J., Ren, X., Menzies, T., & Lyu, M. R.-T. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3709872
“In contrast to current LLM-based test case generation meth-ods [7], [55], [59], [84], [94], [95], STG EN aims to generate test cases to evaluate the time efficiency of code solutions under extreme conditions rigorously.”
Applying Genetic Improvement Techniques for Automated Program Repair of Transpiled Code
Jadhwani, P., Hanna, C., Langdon, W. B., & Petke, J. (2026) · doi:10.1145/3786162.3793234
From C to verifiable Rust: Towards practical migration of code and specifications
Xia, S., Ou, Y., Su, C. J., Guo, Y., Li, Y., & Chen, L. (2026) · Science of Computer Programming · doi:10.1016/j.scico.2026.103535
Not All Local LLMs Are Equal: A Benchmark of Energy and Performance
Cunha, S., Ribeiro, F., Cruz, L., & Saraiva, J. (2026) · IEEE/ACM International Workshop on Green and Sustainable Software (GREENS) · doi:10.1145/3786148.3788630 · Cited by 1
Multi-task Code LLMs: Data Mix or Model Merge?
Zhu, M., Merler, M., Sobolev, B., Krishna, R., Pavuluri, R., & Patterson, S. (2026) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3786181.3788714
LLM-Powered On-Demand Test Suites in Self-Graded Student Programming Assignments
Liu, C. (2026) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3786181.3788709
When to Answer and When to Defer: A Decision Framework for Reliable Code Predictions
Rathnasuriya, R., & Yang, W. M. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786582.3786845
On-the-Fly Input Adaptation for Reliable Code Intelligence
Rathnasuriya, R., & Yang, W. M. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786582.3786844
StackingNet: Collective Inference Across Independent AI Foundation Models
Li, S., Liu, C., Wu, D., Zeng, Z., & Ding, L. (2026) · Advanced Science · doi:10.1002/advs.76488
Wang, M., Cao, H., & Zhao, T. (2026) · Journal of Terminology and Knowledge Engineering · doi:10.3724/jtke-20250007
Prompt Optimization for LLM Code Generation via Reinforcement Learning
Esfahani, A. M., Kahani, N., & Ajila, S. A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30699-9_3
Omnia: Efficient RAG Serving through Speculative Scheduling
Fu, R., Li, S., Xu, Y., Wu, T., Ma, Z., & Shi, J. (2026) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3806645.3807814
“Large Language Models (LLMs) have refined many language-based applications, such as chatbot [1, 43], code generation [28, 32] and text summarization [44].”
VTool: Benchmarking Tool Use Ability in Multi-modal Context
Li, Y., Li, Z., Feng, T., Deng, Y., & ZHANG, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2852-2_48
Do Language Models Prefer Vulnerable Code? A Probabilistic Study of Insecure Code Preference
Melo, R., Reis, S., Catarino, A., & Abreu, R. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00075
Li, L. E., Huang, Y., Shi, J., & Yang, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3435-6_41
Harnessing Hype to Teach Empirical Thinking: An Experience With AI Coding Assistants
Wyrich, M., Peitek, N., Weis, K., & Apel, S. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805785
Evaluating and Improving the Quality of LLM-Generated Code
Melo, G., Pourleyli, J., Caumartin, G., Yang-Smith, C., Costa, D. E., & Abdellatif, A. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3804900
“The discussion will emphasize critical interpretation of metrics, supported by recent empirical studies [2–4], and include basic visualizations within the notebook.”
Revisiting TuRTLe: A Comprehensive Evaluation of LLMs for RTL Generation
Albertí, M., Gutierrez-Gomez, C., García-Gasulla, D., Parisi, E., Ghorab, R. M., Montenegro, O., Homs, B., Moretó, M., & Kestor, G. (2026) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3831369
Exploring the Potential of Large Language Models in Simulink-Stateflow Mutant Generation
Valle, P., Ali, S., & Arrieta, A. (2026) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3793655.3793737 · Cited by 2
“The recent adoption of LLMs in software engineering tasks [19, 22, 23, 26, 32] presents an opportunity to address these domain-specific challenges.”
A Code Ranking Framework with Human-Inspired Agent-Based Tests Generation
Cao, L., Liang, J., Wang, J., & Cai, Y. (2026) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3794763.3794820
Self-Improving Code Generation via Semantic Entropy and Behavioral Consensus
Zhang, H., Cheng, W., & Hu, W. P. (2026) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3794763.3794798 · Cited by 2
“It outperforms both SFT-and DPO-based base-lines using significantly less training data.”
MERIT: A Multi-Signal Repair Framework for Code Translation with Large Language Models
Guo, J. (2026) · International Conference on Big Data & Artificial Intelligence & Software Engineering (ICBASE) · doi:10.1109/icbase70763.2026.11619379
Jeevitha, D., Kasthuri, K., Aishwarya, M., & Madhushree, R. (2026) · International Conference on Computing Methodologies and Communication (ICCMC) · doi:10.1109/iccmc69250.2026.11624731
“In practice, prompt-based generators tend to treat each file or component in isolation; the resulting outputs may pass a syntax check yet lack the cross-file dependency resolution and deployment configuration that a working application demands [1].”
Automated Test Suite Enhancement Using Large Language Models with Few-shot Prompting
Chudic, A., & Çalıklı, G. (2026) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3794763.3794828 · Cited by 1
Guidelines to Prompt Large Language Models for Code Generation: An Empirical Characterization
Midolo, A., Giagnorio, A., Zampetti, F., Tufano, R., Bavota, G., & Penta, M. D. D. (2026) · IEEE International Conference on Program Comprehension · doi:10.1145/3794763.3794819 · Cited by 6
“For our study, we leverage tasks from three benchmarks created for evaluating code generation models in Python: BigCodeBench [17], HumanEval+ [33], and MBPP+ [33].”
InfluentialiBRAHIMOĞLU, N., Demirezen, M. U., & Yıldız, F. (2026) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-026-01490-x · Cited by 1
TextBFGS: A Case-Based Reasoning Approach to Code Optimization via Error-Operator Retrieval
Zhang, Z., Liao, Y., Chen, C., He, J., Wu, D., Yu, Q., Gao, Y., Yang, J., Zhang, K., Zhong, X., et al. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-33865-5_19
“We employ EvalPlus 7 [13] as the evaluation toolkit, which incorporates two standard benchmarks: HumanEval [4] and MBPP [3].”
Test Coverage of Code Changes in AI-Generated Pull Requests
Alves, T., & Teixeira, L. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793618
“Previous work documented latent defects in LLM-generated code that pass functional tests [9] and security vulnerabilities in AI-assisted outputs [10].”
Cooperation makes better: Multi-agent interactive evolution framework in LLM4OR
She, C., Lu, X., Chen, Z., Wang, J., & Zhao, P. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133754
Kim, D. K., Lee, Y.-W., & Jeong, J.-S. (2026) · Applied Sciences · doi:10.3390/app16157733
R-Select: A Robust Multi-Metric Data Selection Approach for Fine-Tuning Large Language Models
Gao, X., Wang, X., Zhu, Y., Liu, Z., He, C., & Wu, L. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817656
“For the Code domain, evaluation is performed on HumanEval [10], LCB (V5) [30], and HumanEval+ [44].”
LLM as Automated Theorem Provers for Map-Reduce Code Generation
Simonov, V. S., & Khairetdinov, M. S. (2026) · International Siberian workshops and tutorials on electron devices and materials · doi:10.1109/edm69524.2026.11632200
“To the best of our knowledge, existing work on LLMs for Python-to-Spark translation focuses exclusively on end-to-end code generation [10], [11], [12].”
Cross-Model Collaboration for Enhancing LLM-Based Code Generation
Huang, J., Ye, W., Sun, W., Feng, Y., & Liu, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3840382
Metamorphic Testing of Multi-Agent LLM Systems: A Trace-Based Behavioral Oracle Framework
Marimuthu, G. (2026) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest70988.2026.00036
“Evaluation of multi-agent LLM systems has primarily focused on task-level benchmarks that measure end-to-end success rates [19].”
FTDI: A budget-aware self-healing framework for LLM multi-agent code generation
Men, S., Lu, Z., Luo, H., & Zheng, Y. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134164
A Dynamic Evaluation Approach to Repository-Level Code Generation Via LLM
Kuang, F., Liu, D., Wang, Y., Huan, H., Zhao, M., Mi, L., & Zheng, X. (2026) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest70988.2026.00019
“EvalPlus [8] strengthens functional evaluation through test-set expansion, and CWEval [9] extends evaluation from functional correctness to security-sensitive behavior.”
Self-citationComplexity Backpressure for AI Coding Agents: A Cross-Model Study on Substantive Software Tasks
Shaw, A., & Beheshti, A. (2026) · IEEE International Conference on Software Services Engineering (SSE) · doi:10.1109/sse72781.2026.00047
Structured Business Logic Injection for LLM-Based Code Generation: What a Design Document Must Carry
Kim, M.-S., Choi, M., Shim, M., & Chung, S.-T. (2026) · Electronics · doi:10.3390/electronics15173884
Opportunities and challenges of generative AI in the research lifecycle
Sadeghi, N., Nakamura, E., Norman, L. J., Okur, A., Ouyang, J. X., Sudre, G. P., & White, T. J. H. (2026) · Aperture Neuro · doi:10.52294/001c.165447
Efficiency Evaluation Framework for LLM-Generated Code
Avagian, D. A., & Ball, A. L. J. (2026) · Pattern Recognition and Image Analysis · doi:10.1134/s1054661826700306
Comprehensive and Realistic Evaluation on the Code Reasoning Capability of LLMs
Liu, C. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3774748.3787667
Comfrey: Mitigating Integration Failures in LLM-enabled Software at Run-Time
Shao, Y., Huang, Y., Zou, J., Shi, Y., Yang, L., Ma, L., Su, T., & Wan, C. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787847 · Cited by 2
LLM-based API Argument Completion with Knowledge-Augmented Prompts
Akram, W., Jiang, Y., Khan, H. A., Jalil, F., & Liu, H. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773158
TACO: Trust Assessment of Large Language Models in Coding Assistance Tasks
Weng, S., Feng, Y., Li, J., Yin, Y., Zhang, Z., Liu, L., & Liu, J. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787835
Self-citationAgent-Based Ensemble Reasoning for Repository-Level Issue Resolution
Tian, Z., Gao, P., Chen, J., & Peng, C. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787761
MioHint: LLM-Assisted Request Mutation for Whitebox REST API Testing
Li, J., Shen, J., Su, Y., & Lyu, M. R.-T. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773228
“It presents an impressive performance at code understanding and generation [22, 34].”
SEBGen: A Self-Evolving Benchmark Generation Framework for Software Engineering Agents
Huang, J., Feng, Z., Xiong, D., Ren, C., & Geng, Y. (2026) · Electronics · doi:10.3390/electronics15204663
CoLSM: Collaborative Large and Small Models for Automatic Software Generation
Wen, Q., Zhao, L., Zhang, X., Zhang, J., Wang, C., & Zhang, P. (2026) · International Conference on Big Data and Information Analytics (BigDIA) · doi:10.1109/bigdia71496.2026.11684680
“Standalone large language models possess strong generalization and high-level logical design capabilities, achieving competitive performance on standard coding benchmarks [8], [9].”
RLOps Pipeline Development with Low-Code and Large Language Models for Industry 4.0
Warnett, S. J., Zdun, U., & Geiger, S. (2026) · IEEE/ACM International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1145/3793653.3793779
Adaptive Stopping in Iterative LLM Loops: Measuring Regret and Recoverable Policy GainPreprint
Zhai, Y., Nie, C., Wang, G., Wang, Z., & Zhou, J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7507225
Paech, A. (2026) · HMD Praxis der Wirtschaftsinformatik · doi:10.1365/s40702-026-01330-8
de Leao, T. M. (2026) · Research Square · doi:10.21203/rs.3.rs-11224002/v1
Memory Energy Efficiency for Edge LLM Inference in AIoT Platforms
Domingues, G. F., Wolff, A. G., Aragão, E. L., Balreira, D. G., Nazar, G. L., & Carro, L. (2026) · IEEE International Conference on Omni-layer Intelligent Systems (COINS) · doi:10.1109/coins69995.2026.11715626
“Large Language Models (LLMs) have become indispensable tools in both industry and academia, demonstrating pro fi - ciency in classical text processing [1] and enabling cutting-edge applications in personalized recommendations, code assistants [2], and healthcare [3].”
Metadata
- Authors
- Liu, J., Xia, C. S., Wang, Y., & Zhang, L.
- ORCID
- Xia, Chunqiu Steven
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 21558-21572
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133198585
- Cited by
- 384
- References
- 0
- Citations by year
- 2026: 143 · 2025: 116 · 2024: 102 · 2023: 23
- FWCI
- 98.69
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2131943, 2141474)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- large language models, code generation, ChatGPT, program correctness, program verification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
