The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks
Open accessAbstract
Deep learning (DL) techniques are gaining more and more attention in the software engineering community. They have been used to support several code-related tasks, such as automatic bug fixing and code comments generation. Recent studies in the Natural Language Processing (NLP) field have shown that the Text-To-Text Transfer Transformer (T5) architecture can achieve state-of-the-art performance for a variety of NLP tasks. The basic idea behind T5 is to first pre-train a model on a large and generic dataset using a self-supervised task (e.g., filling masked words in sentences). Once the model is pre-trained, it is fine-tuned on smaller and specialized datasets, each one related to a specific task (e.g., language translation, sentence classification). In this paper, we empirically investigate how the T5 model performs when pre-trained and fine-tuned to support code-related tasks. We pre-train a T5 model on a dataset composed of natural language English text and source code. Then, we fine-tune such a model by reusing datasets used in four previous works that used DL techniques to: (i) fix bugs, (ii) inject code mutants, (iii) generate assert statements, and (iv) generate code comments. We compared the performance of this single model with the results reported in the four original papers proposing DL-based solutions for those four tasks. We show that our T5 model, exploiting additional data for the self-supervised pre-training phase, can achieve performance improvements over the four baselines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 1×
“commonly adopted in Neural Machine Translation (NMT) [22]–[24], to learn how to automatically fix bugs in Java methods.”
BackgroundBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
Long Short-Term Memory-Networks for Machine Reading
Cheng, J., Dong, L., & Lapata, M. (2016) · arXiv · doi:10.48550/arxiv.1601.06733 · Cited by 1,198
Code completion with statistical language models
Raychev, V., Vechev, M., & Yahav, E. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594321 · Cited by 555
CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., & Brockschmidt, M. (2019) · arXiv · doi:10.48550/arxiv.1909.09436 · Cited by 483
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474
Research works printed but not linked: 57
- 10.1145/2786805.2803196 · DOI
- 10.1145/3106237.3106280 · DOI
- 10.1109/ASE.2013.6693078 · DOI
- 10.1109/ICSE.2013.6606624 · DOI
- 10.1145/1882291.1882315 · DOI · Paper page
- 10.1109/ICPC.2009.5090047 · DOI
- 10.1145/2635868.2635898 · DOI · Paper page
- 10.1145/2591062.2591114 · DOI
- 10.1145/3360585 · DOI · Paper page
- 10.1109/SANER.2019.8668043 · DOI · Paper page
- 10.1016/j.jss.2013.10.019 · doi:10.1016/j.jss.2013.10.019
- graves, 2012, Sequence transduction with recurrent neural networks, CoRR, abs 1211 3711
- boulanger-lewandowski, 2013, Audio chord recognition with recurrent neural networks, ISMIR, 335
- 10.1109/ICSE.2012.6227211 · DOI · Paper page
- 10.1109/TSE.2011.104 · DOI · Paper page
- 10.1145/2813885.2737988 · DOI
- 0, Google colaboraty
- lin, 2004, Rouge: A package for automatic evaluation of summaries, Text Summarization Branches Out, 74
- caswell, 2020, Recent advances in google translate
- 10.1109/TSE.2019.2940179 · DOI · Paper page
- kalchbrenner, 2013, Recurrent continuous translation models, Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 1700
- brody, 2020, Neural edit completion, arXiv preprint arXiv 2005 13209
- sutskever, 2014, Sequence to sequence learning with neural networks, CoRR, abs 1409 3215
- 0, Replication Package
- 10.18653/v1/P16-1195 · DOI · Paper page
- 10.18653/v1/D18-2012 · DOI · Paper page
- srivastava, 2014, Dropout: a simple way to prevent neural networks from overfitting, The Journal of Machine Learning Research, 15, 1929
- ba, 2016, Layer normalization, arXiv preprint arXiv 1607 06450
- 10.18653/v1/D16-1053 · DOI · Paper page
- vaswani, 2017, Attention is all you need, Advances in neural information processing systems, 5998
- allamanis, 2016, A convolutional attention network for extreme summarization of source code, CoRR, abs 1602 3001
- 10.1145/3238147.3238190 · DOI · Paper page
- 10.1145/3379597.3387449 · DOI
- 10.1145/2025113.2025179 · DOI · Paper page
- 10.1109/ASE.2015.36 · DOI · Paper page
- 10.1145/3106237.3106289 · DOI
- 10.1145/3196321.3196330 · DOI
- 10.1109/ICSME.2019.00046 · DOI
- 10.1145/3377811.3380429 · DOI
- karampatsis, 2019, Maybe deep neural networks are the best choice for modeling source code, CoRR, abs 1903 5734
- alon, 2019, Structural language models of code, ArXiv
- hata, 2018, Learning to generate corrective patches using neural machine translation, CoRR, abs 1812 7170
- 10.1145/3338906.3340455 · DOI
- 10.1145/2591062.2591072 · DOI
- tufano, 2019, On learning via neural machine translation, Proceedings of the 41st International Conference on Software Engineering ICSE 2019, 2019, 25
- 10.1109/ICSE.2019.00087 · DOI · Paper page
- 10.1145/2491411.2494584 · DOI
- 10.1145/3196321.3196334 · DOI · Paper page
- 10.1109/ASE.2017.8115626 · DOI · Paper page
- 10.1109/ICSE.2017.13 · DOI
- 10.1109/ICPC.2013.6613830 · DOI · Paper page
- 10.1109/TSE.2015.2465386 · DOI · Paper page
- 10.1145/1985793.1985808 · DOI · Paper page
- 0, Utilizing Fast Testing to Transform Java Development into an Agile Quick Release Low Risk Process
- 10.1145/1297846.1297902 · DOI · Paper page
- 10.1109/ICPC.2011.28 · DOI
- 10.1109/WCRE.2010.13 · DOI · Paper page
Cited by
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
Metadata
- Authors
- Mastropaolo, A., Scalabrino, S., Cooper, N., Nader Palacio, D., Poshyvanyk, D., Oliveto, R., & Bavota, G.
- Venue
- 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)
- Publisher
- IEEE
- Published
- 2021-05
- Pages
- 336-347
- OpenAlex
- W3127853158
- Cited by
- 247
- References
- 13
- FWCI
- 38.49
- Type
- conference paper
- Language
- en
- Open access
- green
- Funders
- European Research Council
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks. https://app.cochat.ai/papers/10.1109/icse43902.2021.00041
- The paper
- Mastropaolo, A., Scalabrino, S., Cooper, N., Nader Palacio, D., Poshyvanyk, D., Oliveto, R., & Bavota, G. (2021). Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks. 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). https://doi.org/10.1109/icse43902.2021.00041
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.