CoChat
Sign Up
PreprintCited by 2Software Engineering ResearchTopic ModelingTeaching and Learning Programming

FullStack Bench: Evaluating LLMs as Full Stack Coders

Open access
Bytedance-Seed-Foundation-Code-Team, :, Cheng, Yao, Chen, Jianfeng, Chen, Jie, Chen, Li, Chen, Liyu, Chen, Wentao +48 more·2024·arXiv
DOI 10.48550/arxiv.2412.00535arXiv 2412.00535
Open this paper in CoChat — free accountOpen PDF

Abstract

As the capabilities of code large language models (LLMs) continue to expand, their applications across diverse code intelligence domains are rapidly increasing. However, most existing datasets only evaluate limited application domains. To address this gap, we have developed a comprehensive code evaluation dataset FullStack Bench focusing on full-stack programming, which encompasses a wide range of application domains (e.g., basic programming, data analysis, software engineering, mathematics, and machine learning). Besides, to assess multilingual programming capabilities, in FullStack Bench, we design real-world instructions and corresponding unit test cases from 16 widely-used programming languages to reflect real-world usage scenarios rather than simple translations. Moreover, we also release an effective code sandbox execution tool (i.e., SandboxFusion) supporting various programming languages and packages to evaluate the performance of our FullStack Bench efficiently. Comprehensive experimental results on our FullStack Bench demonstrate the necessity and effectiveness of our FullStack Bench and SandboxFusion.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2412.00535

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Repository-level evaluations reveal cross-file integration and dependency breaks.

    “Repository-level evaluations reveal similar cross-file integration and dependency breaks [10, 22].”

    From Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification (2025) · cites this paper as [22]

    The Code Benchmark section mentions repository-level benchmarks, but does not report that they reveal cross-file integration or dependency breaks. Searches for those failure types found only discussion of sandbox execution and package-version compatibility, not the claimed evaluation finding. The cited passages provide no page numbers. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

  • Constraint-reconciliation decay occurs: as sessions lengthen and dependencies accumulate, success rates fall.

    “This results in constraintreconciliation decay : as sessions lengthen and dependencies accumulate, success rates fall.”

    From Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification (2025) · cites this paper as [22]

    The Implementation Details section evaluates each problem as a separate user prompt, and §4.5 compares performance by problem difficulty. Neither passage addresses whether success rates fall as sessions lengthen or dependencies accumulate; searches for session length, sequential dependencies, and constraint reconciliation found no such analysis. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 28

  1. B. Athiwaratkun, S. K. Gouda, Z. Wang, X. Li, Y. Tian, M. Tan, W. U. Ahmad, S. Wang, Q. Sun, M. Shang, S. K. Gonugondla, H. Ding, V. Kumar, N. Fulton, A. Farahani, S. Jain, R. Giaquinto, H. Qian, M. K. Ramanathan, R. Nallapati, B. Ray, P. Bhatia, S. Sengupta, D. Roth, and B. Xiang. Multi-lingual evaluation of code generation models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=Bo7eeXm6An8.
  2. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021b. URL https://arxiv.org/abs/2108.07732. · arXiv
  3. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code, 2021a. URL https: //arxiv.org/abs/2107.03374.
  4. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
  5. D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, S. Yih, L. Zettlemoyer, and M. Lewis. Incoder: A generative model for code infilling and synthesis. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/f orum?id=hQwb-lbM6EL.
  6. L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: Program- aided language models. In International Conference on Machine Learning, pages 10764-10799.
  7. PMLR, 2023.
  8. D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li, et al. Deepseek- coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024b. · arXiv
  9. D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Measuring coding challenge competence with APPS. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openreview.net/forum?id=sD93GOzH3i5.
  10. S. Huang, T. Cheng, J. Klein Liu, J. Hao, L. Song, Y. Xu, J. Yang, J. Liu, C. Zhang, L. Chai, et al. Opencoder: The open cookbook for top-tier code large language models. arXiv e-prints, pages arXiv-2411, 2024a.
  11. S. Huang, T. Cheng, J. K. Liu, J. Hao, L. Song, Y. Xu, J. Yang, J. H. Liu, C. Zhang, L. Chai, R. Yuan, Z. Zhang, J. Fu, Q. Liu, G. Zhang, Z. Wang, Y. Qi, Y. Xu, and W. Chu. Opencoder: The open cookbook for top-tier code large language models. In arXiv preprint, 2024b.
  12. F. Huq, M. Hasan, M. M. A. Haque, S. Mahbub, A. Iqbal, and T. Ahmed. Review4repair: Code review aided automatic program repairing. Information and Software Technology, 143:106765, 2022.
  13. M. Jiao, T. Yu, X. Li, G. Qiu, X. Gu, and B. Shen. On the evaluation of neural code translation: Taxonomy and benchmark. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1529-1541. IEEE, 2023.
  14. W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611-626, 2023.
  15. Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W. Yih, D. Fried, S. I. Wang, and T. Yu. DS-1000: A natural and reliable benchmark for data science code generation. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 18319-18345. PMLR, 2023. URL https: //proceedings.mlr.press/v202/lai23b.html.
  16. Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gi- meno, A. D. Lago, et al. Competition-level code generation with alphacode. ArXiv preprint, abs/2203.07814, 2022. URL https://arxiv.org/abs/2203.07814. · arXiv
  17. J. Liu, C. S. Xia, Y. Wang, and L. Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 -16, 2023, 2023a. URL http://papers.nips.cc /paper_files/paper/2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstrac t-Conference.html.
  18. J. Liu, K. Deng, C. Liu, J. Yang, S. Liu, H. Zhu, P. Zhao, L. Chai, Y. Wu, K. Jin, G. Zhang, Z. M. Wang, G. Zhang, B. Xiang, W. Su, and B. Zheng. M2rc-eval: Massively multilingual repository-level code completion evaluation. In arXiv preprint, 2024a.
  19. S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. GONG, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. LIU. CodeXGLUE: A machine learning bench- mark dataset for code understanding and generation. In Thirty-fifth Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https://openreview.net/forum?id=6lE4dQXaUcb.
  20. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhu- moye, Y. Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
  21. E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id= iaYcJKpY2B_.
  22. C. Richter and H. Wehrheim. Tssb-3m: Mining single statement bugs at massive scale. In Proceedings of the 19th International Conference on Mining Software Repositories, pages 418-422, 2022.
  23. D. Shrivastava, H. Larochelle, and D. Tarlow. Repository-level prompt generation for large language models of code. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 31693-31715. PMLR, 23-29 Jul 2023b. URL https://proceedings.mlr.press/v202/shrivastava23a.html.
  24. T. Sun, L. Chai, Y. Y. Jian Yang, H. Guo, J. Liu, B. Wang, L. Yang, and Z. Li. Unicoder: Scaling code large language model via universal code. ACL, 2024.
  25. F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn. A systematic evaluation of large language models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1-10, 2022.
  26. H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, Q. Wang, and T. Xie. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pages 1-12, 2024.
  27. K. Zhu, Q. Zang, S. Jia, S. Wu, F. Fang, Y. Li, S. Guo, T. Zheng, B. Li, H. Wu, et al. Lime-m: Less is more for evaluation of mllms. arXiv preprint arXiv:2409.06851, 2024a. · arXiv
  28. Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y. Wu, Y. Li, H. Gao, S. Ma, et al. Deepseek- coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931, 2024b. Series Name Model Name (Size) Llama CodeLlama-7B-Instruct (7B) CodeLlama-13B-Instruct (13B) CodeLlama-34B-Instruct (34B) Qwen2.5 Qwen2.5-Coder-1.5B-Instruct (1.5B) Qwen2.5-Coder-7B-Instruct (7B) Qwen2.5-Coder-14B-Instruct (14B) Qwen2.5-Coder-32B-Instruct (32B) DeepSeek v1 DeepSeek-Coder-1.3B-Instruct (1.3B) DeepSeek-Coder-6.7B-Instruct (6.7B) DeepSeekCoder-33B-Instruct (33B) DeepSeek v2 DeepSeek-Coder-7B-Instruct-v1.5 (7B) DeepSeekCoder-v2-Lite-Instruct (16B) DeepSeekCoder-v2-Instruct (236B) OpenCoder OpenCoder-1.5B-Instruct (1.5B) OpenCoder-8B-Instruct (8B) Table 4. Model Series discussed in Section 4.4. Open-Sourced Model Model Link CodeQwen1.5-7B-Instruct https://hf.co/Qwen/CodeQwen1.5-7B-Instruct Qwen2.5-Coder-1.5B-Instruct https://hf.co/Qwen/Qwen2.5-Coder-1.5B-Instruct Qwen2.5-Coder-7B-Instruct https://hf.co/Qwen/Qwen2.5-Coder-7B-Instruct Qwen2.5-Coder-14B-Instruct https://hf.co/Qwen/Qwen2.5-Coder-14B-Instruct Qwen2.5-Coder-32B-Instruct https://hf.co/Qwen/Qwen2.5-Coder-32B-Instruct Qwen2.5-72B-Instruct https://hf.co/Qwen/Qwen2.5-72B-Instruct CodeLlama-7B-Instruct https://hf.co/meta-llama/CodeLlama-7b-Instruct-hf CodeLlama-34B-Instruct https://hf.co/meta-llama/CodeLlama-34b-Instruct-hf CodeLlama-13B-Instruct https://hf.co/meta-llama/CodeLlama-13b-Instruct-hf Llama3.1-70B-Instruct https://hf.co/meta-llama/Llama-3.1-70B-Instruct DeepSeek-Coder-1.3B-Instruct https://hf.co/deepseek-ai/deepseek-coder-1.3b-instruct DeepSeek-Coder-6.7B-Instruct https://hf.co/deepseek-ai/deepseek-coder-6.7b-instruct DeepSeek-Coder-33B-Instruct https://hf.co/deepseek-ai/deepseek-coder-33b-instruct DeepSeek-Coder-7B-Instruct-v1.5 https://hf.co/deepseek-ai/deepseek-coder-7b-instruct-v1. · arXiv

Cited by

Sorted by citations

Related

Authors
Bytedance-Seed-Foundation-Code-Team, :, Cheng, Y., Chen, J., Chen, J., Chen, L., Chen, L., Chen, W., Chen, Z., Geng, S., Li, A., Li, B., Li, B., Li, L., Liu, B., Liu, J., Liu, K., Liu, Q., Liu, S., Liu, S., Liu, T., Liu, T., Liu, Y., Long, R., Mai, J., Ning, G., Peng, Z. Y., Shen, K., Su, J., Su, J., Sun, T., Sun, Y., Tao, Y., Wang, G., Wang, S., Wang, X., Wang, Y., Wang, Z., Xia, J., Xiang, L., Xiao, X., Xiao, Y., Xi, C., Xin, S., Xu, J., Xu, S., Yang, H., Yang, J., Yang, Y., Yuan, J., Zhang, J., Zhang, Y., Zhang, Y., Zheng, S., Zhu, H., & Zhu, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2412.00535
arXiv
2412.00535
OpenAlex
W4405033162
Cited by
2
References
49
Citations by year
2025: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
code large language models, multilingual programming, unit test generation, code generation evaluation, data analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review