TreeGen: A Tree-Based Transformer Architecture for Code Generation
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
A code generation system generates programming language code based on an input natural language description. State-of-the-art approaches rely on neural networks for code generation. However, these code generators suffer from two problems. One is the long dependency problem, where a code element often depends on another far-away code element. A variable reference, for example, depends on its definition, which may appear quite a few lines before. The other problem is structure modeling, as programs contain rich structural information. In this paper, we propose a novel tree-based neural architecture, TreeGen, for code generation. TreeGen uses the attention mechanism of Transformers to alleviate the long-dependency problem, and introduces a novel AST reader (encoder) to incorporate grammar rules and AST structures into the network. We evaluated TreeGen on a Python benchmark, HearthStone, and two semantic parsing benchmarks, ATIS and GEO. TreeGen outperformed the previous state-of-the-art approach by 4.5 percentage points on HearthStone, and achieved the best accuracy among neural network-based approaches on ATIS (89.1%) and GEO (89.6%). We also conducted an ablation test to better understand each component of our model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Xception: Deep Learning with Depthwise Separable Convolutions
Chollet, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.195 · Cited by 20,380
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
Get To The Point: Summarization with Pointer-Generator Networks
See, A., Liu, P. J., & Manning, C. D. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1099 · Cited by 3,775
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537
Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
Shazeer, N., & Stern, M. (2018) · arXiv · doi:10.48550/arxiv.1804.04235 · Cited by 1,253
Language to Logical Form with Neural Attention
Dong, L., & Lapata, M. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1004 · Cited by 700
A Syntactic Neural Model for General-Purpose Code Generation
Yin, P., & Neubig, G. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1041 · Cited by 623
A Syntactic Neural Model for General-Purpose Code Generation
Yin, P., & Neubig, G. (2017) · arXiv · doi:10.48550/arxiv.1704.01696 · Cited by 585
Dehghani, M., Gouws, S., Vinyals, O., Uszkoreit, J., & Kaiser, Ł. (2018) · arXiv · doi:10.48550/arxiv.1807.03819 · Cited by 467
Cited by
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599790 · Cited by 563
“From LISPbased pioneering deductive synthesis approaches [31, 37] tomodern program synthesis systems [23, 30], to end-to-end code generation via deep neural networks [19, 32, 33], tremendous efforts have been made to enable machines to automatically write correct programs as part of the quest to artificial general int…”
BackgroundHam, T. J., Lee, Y., Seo, S. H., Kim, S., Choi, H., Jung, S. J., & Lee, J. W. (2021) · 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA) · doi:10.1109/isca52012.2021.00060 · Cited by 192
Self-Planning Code Generation with Large Language Models
Jiang, X., Dong, Y., Wang, L., Fang, Z., Shang, Q., Li, G., Jin, Z., & Jiao, W. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3672456 · Cited by 158
Investigating Code Generation Performance of ChatGPT with Crowdsourcing Social Data
Feng, Y., Vanam, S., Cherukupally, M., Zheng, W., Qiu, M., & Chen, H. (2023) · 2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac57700.2023.00117 · Cited by 117
Siddiq, M. L., & Santos, J. C. S. (2022) · Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security · doi:10.1145/3549035.3561184 · Cited by 101
Yetiştiren, B., Özsoy, I., Ayerdem, M., & Tüzün, E. (2023) · arXiv · doi:10.48550/arxiv.2304.10778 · Cited by 75
SkCoder: A Sketch-based Approach for Automatic Code Generation
Li, J., Li, Y., Li, G., Jin, Z., Hao, Y., & Hu, X. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00179 · Cited by 64
An Empirical Study of Code Smells in Transformer-based Code Generation Techniques
Siddiq, M. L., Majumder, S. H., Mim, M. R., Jajodia, S., & Santos, J. C. S. (2022) · 2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam55253.2022.00014 · Cited by 70
FIRA: fi ne-grained g ra ph-based code change representation for automated commit message generation
Dong, J., Lou, Y., Zhu, Q., Sun, Z., Li, Z., Zhang, W., & Hao, D. (2022) · Proceedings of the 44th International Conference on Software Engineering · doi:10.1145/3510003.3510069 · Cited by 61
Self-citationAn Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities
Yang, Z., Chen, S., Gao, C., Li, Z., Hu, X., Liu, K., & Xia, X. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3717061 · Cited by 37
Large Language Model-Aware In-Context Learning for Code Generation
Li, J., Li, G., Tao, C., Li, J., Zhang, H., Liu, F., & Jin, Z. (2023) · arXiv · doi:10.48550/arxiv.2310.09748 · Cited by 32
Large Language Model-Aware In-Context Learning for Code Generation
Li, J., Tao, C., Li, J., Li, G., Jin, Z., Zhang, H., Fang, Z., & Liu, F. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3715908 · Cited by 32
Liao, D., Pan, S., Sun, X., Ren, X., Huang, Q., Xing, Z., Jin, H., & Li, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3486195 · Cited by 23
Metadata
- Authors
- Sun, Z., Zhu, Q., Xiong, Y., Sun, Y., Mou, L., & Zhang, L.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2020-04-03
- Volume
- 34
- Issue
- 05
- Pages
- 8984-8991
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2997847174
- Cited by
- 167
- References
- 13
- FWCI
- 20.84
- Type
- conference paper
- Language
- en
- Open access
- diamond
- Licence
- www.aaai.org/ (unspecified)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.