Quality Estimation & Interpretability for Code Translation
Open accessAbstract
Recently, the automated translation of source code from one programming language to another by using automatic approaches inspired by Neural Machine Translation (NMT) methods for natural languages has come under study. However, such approaches suffer from the same problem as previous NMT approaches on natural languages, viz. the lack of an ability to estimate and evaluate the quality of the translations; and consequently ascribe some measure of interpretability to the model's choices. In this paper, we attempt to estimate the quality of source code translations built on top of the TransCoder model. We consider the code translation task as an analog of machine translation (MT) for natural languages, with some added caveats. We present our main motivation from a user study built around code translation; and present a technique that correlates the confidences generated by that model to lint errors in the translated code. We conclude with some observations on these correlations, and some ideas for future work.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Indicating model uncertainty and an explanation for that uncertainty can help users disentangle the role of their prompting and output evaluation ability from that of models' capabilities, further supporting confidence adjustment.
“Finally, indicating model uncertainty [19,174,198], for example, by means of color-coding [167], and an explanation for that uncertainty [5,190], can also help users disentangle the role of their prompting and output evaluation ability from that of models' capabilities, further supporting confidence adjustment [197].”
From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [5]
Section 2.1.2 (p. 2) reports that users struggled to understand when and why the code-translation model lacked confidence, and Section 4.2 (p. 4) identifies a need for interpretable confidence values. This supports the cited point about explaining uncertainty, but the paper does not show that such explanations help users separate their prompting or output-evaluation ability from model capability. (LLM, conf=0.82)
Experimental — accuracy not yet measuredAgarwal et al. demonstrated how to derive line-level confidences by aggregating token probabilities.
“by Agarwal et al. [2], which demonstrates how to derive line-level confidences by aggregating token probabilities.”
From Investigating Explainability of Generative AI for Code through Scenario-based Design (2022) · cites this paper as [2]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
Cited by 3
Research works printed but not linked: 19
- Allamanis, M., Barr, E., and Sutton, C. Learning natural coding conventions. ArXiv , abs/1402.4182, 2014. · arXiv
- Allamanis, M., Barr, E. T., Devanbu, P., and Sutton, C. A survey of machine learning for big code and naturalness. ACM Computing Surveys (CSUR) , 51(4):1-37, 2018.
- Chen, X., Liu, C., and Song, D. Tree-to-tree neural networks for program translation. In Advances in neural information processing systems , pp. 2547-2557, 2018.
- Fomicheva, M., Sun, S., Yankovskaya, L., Blain, F., Guzm´ an, F., Fishel, M., Aletras, N., Chaudhary, V., and Specia, L. Unsupervised quality estimation for neural machine translation. arXiv preprint arXiv:2005.10608 , 2020. · arXiv
- Garg, A. and Agarwal, M. Machine translation: A literature review. arXiv preprint arXiv:1901.01122 , 2018. · arXiv
- Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. arXiv preprint arXiv:1706.04599 , 2017. · arXiv
- Jing, Y., Yang, Y ., Feng, Z., Ye, J., and Song, M. Neural style transfer: A review. IEEE Transactions on Visualization and Computer Graphics , 26:3365-3385, 2020.
- Karaivanov, S., Raychev, V., and Vechev, M. Phrase-based statistical translation of programming languages. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software , pp. 173-184, 2014.
- Kepler, F., Tr´ enous, J., Treviso, M., Vera, M., G´ ois, A., Farajian, M. A., Lopes, A. V., and Martins, A. F. Unbabel's participation in the wmt19 translation quality estimation shared task. arXiv preprint arXiv:1907.10352 , 2019. · arXiv
- Khadka, R., Saeidi, A., Idu, A., Hage, J., and Jansen, S. Legacy to soa evolution: A systematic bibliography, 2012. URL https://www.academia.edu/download/ 30759311/LegacytoSOA-TR2.pdf .
- Kim, H., Lee, J.-H., and Na, S.-H. Predictor-estimator using multilevel task learning with stack propagation for neural quality estimation. In Proceedings of the Second Conference on Machine Translation , pp. 562-568, 2017.
- Nguyen, A. T., Nguyen, T. T., and Nguyen, T. N. Lexical statistical machine translation for language migration. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering , pp. 651-654, 2013.
- Oda, Y., Fudaba, H., Neubig, G., Hata, H., Sakti, S., Toda, T., and Nakamura, S. Learning to generate pseudo-code from source code using statistical machine translation (t). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) , pp. 574-584. IEEE, 2015.
- Roziere, B., Lachaux, M.-A., Chanussot, L., and Lample, G. Unsupervised translation of programming languages. Advances in Neural Information Processing Systems , 33, 2020.
- Tate, R. F. Correlation between a discrete and a continuous variable. point-biserial correlation. The Annals of mathematical statistics , 25(3):603-607, 1954.
- Toral, A., Castilho, S., Hu, K., and Way, A. Attaining the unattainable? reassessing claims of human parity in neural machine translation. arXiv preprint arXiv:1808.10432 , 2018. · arXiv
- Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C. J., Polat, ˙ I., Feng, Y., Moore, E. W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E. A., Harris, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa, F., van Mulbregt, P., and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods , 17:261-272, 2020. doi: 10.1038/s41592-019-0686-2. · DOI · Paper page
- Wang, J., Fan, K., Li, B., Zhou, F., Chen, B., Shi, Y., and Si, L. Alibaba submission for wmt18 quality estimation task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers , pp. 809-815, 2018.
- Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Martinez, F., Agarwal, M., and Talamadupula, K. Perfection not required? human-ai partnerships in code translation. In 26th International Conference on Intelligent User Interfaces , IUI '21, pp. 402-412, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450380171. doi: 10.1145/ 3397481.3450656. URL https://doi.org/10. 1145/3397481.3450656 .
- Authors
- Agarwal, M., Talamadupula, K., Houde, S., Martinez, F., Muller, M., Richards, J., Ross, S., & Weisz, J. D.
- Affiliations
- IBM (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2012.07581
- OpenAlex
- W4287557021
- Cited by
- 3
- References
- 0
- Citations by year
- 2025: 1 · 2024: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- program translation, source code translation, translation quality estimation, model interpretability, transcoder, confidence calibration, user study, neural machine translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
