The DOI matches its Crossref record; 188 of the 188 references it cites were found in a registry or an index.
A Survey of Machine Learning for Big Code and Naturalness
Open accessAbstract
Research at the intersection of machine learning, programming languages, and software engineering has recently taken important steps in proposing learnable probabilistic models of source code that exploit the abundance of patterns of code. In this article, we survey this work. We contrast programming languages against natural languages and discuss how these similarities and differences drive the design of probabilistic models. We present a taxonomy based on the underlying design principles of each model and use it to navigate the literature. Then, we review how researchers have adapted these models to application areas and discuss cross-cutting and application-specific challenges and opportunities.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Data Mining: Practical Machine Learning Tools and Techniques
Witten, I. H., Frank, E., & Hall, M. A. (2011) · Elsevier eBooks · doi:10.1016/c2009-0-19715-5 · Cited by 25,543
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Chandola, V., Banerjee, A., & Kumar, V. (2009) · ACM Computing Surveys · doi:10.1145/1541880.1541882 · Cited by 11,687
(2016) · Cited by 11,679
Distributed Representations of Sentences and DocumentsPreprint
Le, Q. V., & Mikolov, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.4053 · Cited by 7,905
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merriënboer, B., Bahdanau, D., & Bengio, Y. (2014) · SSST@EMNLP · doi:10.3115/v1/w14-4012 · Cited by 7,251
Blei, D. M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133826 · Cited by 5,516
Research works printed but not linked: 68
- Alemi Alex A., 2016, Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS’16)
- Allamanis Miltiadis, 2018, Proceedings of the International Conference on Learning Representations (ICLR’18)
- Allamanis Miltiadis, 2017, Proceedings of the International Conference on Machine Learning (ICML’17)
- Allamanis Miltiadis, 2016, Proceedings of the International Conference on Machine Learning (ICML’16)
- 10.5555/2487085.2487127 · DOI
- Allamanis Miltiadis, 2015, Proceedings of the International Conference on Machine Learning (ICML’15)
- Bahdanau Dzmitry, 2015, Proceedings of the International Conference on Learning Representations (ICLR’15)
- Balog Matej, 2017, Proceedings of the International Conference on Learning Representations (ICLR’17)
- Miceli Barone Antonio Valerio, 2017, Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers) 2
- Bhatia Sahil, 2018, Proceedings of the International Conference on Software Engineering (ICSE’18)
- Bielik Pavol, 2015, Proceedings of the LIPIcs-Leibniz International Proceedings in Informatics.
- Bielik Pavol, 2016, Proceedings of the International Conference on Machine Learning (ICML’16)
- Raymond P., Proceedings of the International Conference on Software Engineering (ICSE’12)
- Deng Yuntian, Proceedings of the International Conference on Machine Learning (ICML’17)
- 10.5555/2819009.2819097 · DOI
- Devlin Jacob, 2017, Proceedings of the International Conference on Machine Learning (ICML’17)
- Dyer Robert, Proceedings of the International Conference on Software Engineering (ICSE’13)
- Ellis Kevin, 2017, Tenenbaum
- Ernst Michael D., 2017, Proceedings of the LIPIcs-Leibniz International Proceedings in Informatics.
- Feser John K., 2017, InProceedings of the International Conference on Learning Representations (ICLR’17)
- Foundation Eclipse, 2017, Retrieved
- 10.5555/2819009.2819143 · DOI
- Goodfellow Ian, Deep Learning
- Gordon Andrew D., Proceedings of the International Conference on Software Engineering (ICSE’14)
- Gotel Orlena, 2012, Andrea Zisman, Alexander Egyed, Paul Grünbacher, Alex Dekhtyar, Giuliano Antoniol, Jonathan Maletic, and Patrick Mäder.
- Gupta Rahul, 2017, Proceedings of the Conference of Artificial Intelligence (AAAI’17)
- 10.5555/2820518.2820539 · DOI
- Hinton G. E., Parallel Distributed Processing: Explorations in the Microstructure of Cognition, 1
- 10.5555/3155562.3155583 · DOI
- Johnson Daniel D., 2016, Proceedings of the International Conference on Learning Representations (ICLR’16)
- Dan Jurafsky. 2000. Speech 8 Language Processing (3rd. ed.). Pearson Education. Dan Jurafsky. 2000. Speech 8 Language Processing (3rd. ed.). Pearson Education.
- Koschke Rainer, 2007, Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum für Informatik.
- Kremenek Ted, Proceedings of the International Joint Conference on Artifical intelligence (IJCAI’07)
- Kushman Nate, 2013, Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT’13)
- Quoc, Proceedings of the International Conference on Machine Learning (ICML’14)
- Levy Dor, 2017, Proceedings of the International Conference on Machine Learning (ICML’17)
- Li Yujia, 2016, Proceedings of the International Conference on Learning Representations (ICLR’16)
- Liang Percy, 2010, Proceedings of the International Conference on Machine Learning (ICML’10)
- Lin Xi Victoria, Proceedings of the International Conference on Language Resources and Evaluation.
- Loos Sarah M., 2017, Proceedings of the International Conference on Logic for Programming Artificial Intelligence and Reasoning (LPAR’17)
- Maddison Chris, 2014, Proceedings of the International Conference on Machine Learning (ICML’14)
- Menon Aditya, 2013, Proceedings of the International Conference on Machine Learning (ICML’13)
- Mens Kim, Recommendation Systems in Software Engineering
- Mou Lili, 2016, Proceedings of the Conference of Artificial Intelligence (AAAI’16)
- Movshovitz-Attias Dana, Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL’13)
- Movshovitz-Attias Dana, 2015, Cohen
- Murali Vijayaraghavan, 2018, Proceedings of the International Conference on Learning Representations (ICLR).
- Neelakantan Arvind, 2015, Proceedings of the International Conference on Learning Representations (ICLR’15)
- Nguyen Anh Tuan, Proceedings of the International Conference on Software Engineering (ICSE’15)
- Nguyen Anh Tuan, Proceedings of the International Symposium on Foundations of Software Engineering (FSE’13)
- Nguyen Anh T., Proceedings of the International Conference on Automated Software Engineering (ASE’15)
- Nguyen Trong Duc, Proceedings of the International Conference on Software Engineering (ICSE’16)
- Nguyen Trong Duc, Proceedings of the International Conference on Software Engineering (ICSE’17)
- Nguyen Tung Thanh, Proceedings of the Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering (ESEC/FSE’13)
- Omar Cyrus, 2017, Jonathan Aldrich, and Matthew A. Hammer.
- Parisotto Emilio, 2017, Proceedings of the International Conference on Learning Representations (ICLR’17)
- Pham Hung Viet, 2016, Proceedings of the International Conference on Software Engineering (ICSE’16)
- Piech Chris, Proceedings of the International Conference on Machine Learning (ICML’15)
- 10.5555/1667583.1667599 · DOI
- Reed Scott, 2016, Proceedings of the International Conference on Learning Representations (ICLR’16)
- Riedel Sebastian, 2017, Proceedings of the International Conference on Machine Learning (ICML’17)
- Rocktäschel Tim, 2017, Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS’17)
- 10.18653/v1/P16-1162 · DOI · Paper page
- Sutskever Ilya, Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS’14)
- Xin Wang Chang Liu Richard Shin Joseph E. Gonzalez and Dawn Song. 2016. Neural Code Completion. Retrieved from https://openreview.net/pdf?id=rJbPBt9lg. Xin Wang Chang Liu Richard Shin Joseph E. Gonzalez and Dawn Song. 2016. Neural Code Completion. Retrieved from https://openreview.net/pdf?id=rJbPBt9lg.
- 10.5555/2820518.2820559 · DOI
- Witten Ian H., 2016, Pal
- Zheng Alice X., 2003, Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS’03)
Cited by
Sorted by citations
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
“Prior work exists on evaluating readability (Allamanis et al., 2018) and security (Pearce et al.”
BackgroundSelf-citationHu, X., Li, G., Xia, X., Lo, D., & Jin, Z. G. (2018) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3196321.3196334 · Cited by 709
Automated Vulnerability Detection in Source Code Using Deep Representation Learning
Russell, R. L., Kim, L. Y., Hamilton, L., Lazovich, T., Harer, J., Özdemir, O., Ellingwood, P. M., & McConley, M. W. (2018) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2018.00120 · Cited by 758
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
IntelliCode compose: code generation using transformer
Svyatkovskiy, A., Deng, S. K., Fu, S., & Sundaresan, N. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417058 · Cited by 474
Metadata
- Authors
- Allamanis, M., Barr, E. T., Devanbu, P., & Sutton, C.
- Affiliations
- Microsoft Research (United Kingdom) (GB)
- University College London (GB)
- University of California, Davis, California, USA
- The Alan Turing Institute (GB)
- Venue
- ACM Computing Surveys
- Published
- 2018-07-31
- Volume
- 51
- Issue
- 4
- Pages
- 1-37
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2754629507
- Cited by
- 150
- References
- 186
- Citations by year
- 2026: 27 · 2025: 11 · 2024: 7 · 2023: 33 · 2022: 19 · 2021: 19 · 2020: 10 · 2019: 15 · 2018: 8
- FWCI
- 20.98
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Computing Surveys (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Engineering and Physical Sciences Research Council (EP/K024043/1, EP/P005659/1, and EP/P005314/1)
- National Science Foundation (1414172)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Logic, programming, and type systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- probabilistic programming languages, naturalness, software engineering applications
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
