CoChat
Sign Up
Cited by 161Software Engineering ResearchLogic, programming, and type systemsNatural Language Processing Techniques

The DOI matches its Crossref record; 188 of the 188 references it cites were found in a registry or an index.

A Survey of Machine Learning for Big Code and Naturalness

Open access
Allamanis, Miltiadis, Barr, Earl T., Devanbu, Premkumar, Sutton, Charles·2019·ACM Computing Surveys
DOI 10.1145/3212695
Open this paper in CoChat — free accountOpen PDF

Abstract

Research at the intersection of machine learning, programming languages, and software engineering has recently taken important steps in proposing learnable probabilistic models of source code that exploit the abundance of patterns of code. In this article, we survey this work. We contrast programming languages against natural languages and discuss how these similarities and differences drive the design of probabilistic models. We present a taxonomy based on the underlying design principles of each model and use it to navigate the literature. Then, we review how researchers have adapted these models to application areas and discuss cross-cutting and application-specific challenges and opportunities.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3212695

Cites

188 of 188 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 186 references

Research works printed but not linked: 68

  1. Alemi Alex A., 2016, Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS’16)
  2. Allamanis Miltiadis, 2018, Proceedings of the International Conference on Learning Representations (ICLR’18)
  3. Allamanis Miltiadis, 2017, Proceedings of the International Conference on Machine Learning (ICML’17)
  4. Allamanis Miltiadis, 2016, Proceedings of the International Conference on Machine Learning (ICML’16)
  5. 10.5555/2487085.2487127 · DOI
  6. Allamanis Miltiadis, 2015, Proceedings of the International Conference on Machine Learning (ICML’15)
  7. Bahdanau Dzmitry, 2015, Proceedings of the International Conference on Learning Representations (ICLR’15)
  8. Balog Matej, 2017, Proceedings of the International Conference on Learning Representations (ICLR’17)
  9. Miceli Barone Antonio Valerio, 2017, Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers) 2
  10. Bhatia Sahil, 2018, Proceedings of the International Conference on Software Engineering (ICSE’18)
  11. Bielik Pavol, 2015, Proceedings of the LIPIcs-Leibniz International Proceedings in Informatics.
  12. Bielik Pavol, 2016, Proceedings of the International Conference on Machine Learning (ICML’16)
  13. Raymond P., Proceedings of the International Conference on Software Engineering (ICSE’12)
  14. Deng Yuntian, Proceedings of the International Conference on Machine Learning (ICML’17)
  15. 10.5555/2819009.2819097 · DOI
  16. Devlin Jacob, 2017, Proceedings of the International Conference on Machine Learning (ICML’17)
  17. Dyer Robert, Proceedings of the International Conference on Software Engineering (ICSE’13)
  18. Ellis Kevin, 2017, Tenenbaum
  19. Ernst Michael D., 2017, Proceedings of the LIPIcs-Leibniz International Proceedings in Informatics.
  20. Feser John K., 2017, InProceedings of the International Conference on Learning Representations (ICLR’17)
  21. Foundation Eclipse, 2017, Retrieved
  22. 10.5555/2819009.2819143 · DOI
  23. Goodfellow Ian, Deep Learning
  24. Gordon Andrew D., Proceedings of the International Conference on Software Engineering (ICSE’14)
  25. Gotel Orlena, 2012, Andrea Zisman, Alexander Egyed, Paul Grünbacher, Alex Dekhtyar, Giuliano Antoniol, Jonathan Maletic, and Patrick Mäder.
  26. Gupta Rahul, 2017, Proceedings of the Conference of Artificial Intelligence (AAAI’17)
  27. 10.5555/2820518.2820539 · DOI
  28. Hinton G. E., Parallel Distributed Processing: Explorations in the Microstructure of Cognition, 1
  29. 10.5555/3155562.3155583 · DOI
  30. Johnson Daniel D., 2016, Proceedings of the International Conference on Learning Representations (ICLR’16)
  31. Dan Jurafsky. 2000. Speech 8 Language Processing (3rd. ed.). Pearson Education. Dan Jurafsky. 2000. Speech 8 Language Processing (3rd. ed.). Pearson Education.
  32. Koschke Rainer, 2007, Dagstuhl Seminar Proceedings. Schloss Dagstuhl-Leibniz-Zentrum für Informatik.
  33. Kremenek Ted, Proceedings of the International Joint Conference on Artifical intelligence (IJCAI’07)
  34. Kushman Nate, 2013, Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT’13)
  35. Quoc, Proceedings of the International Conference on Machine Learning (ICML’14)
  36. Levy Dor, 2017, Proceedings of the International Conference on Machine Learning (ICML’17)
  37. Li Yujia, 2016, Proceedings of the International Conference on Learning Representations (ICLR’16)
  38. Liang Percy, 2010, Proceedings of the International Conference on Machine Learning (ICML’10)
  39. Lin Xi Victoria, Proceedings of the International Conference on Language Resources and Evaluation.
  40. Loos Sarah M., 2017, Proceedings of the International Conference on Logic for Programming Artificial Intelligence and Reasoning (LPAR’17)
  41. Maddison Chris, 2014, Proceedings of the International Conference on Machine Learning (ICML’14)
  42. Menon Aditya, 2013, Proceedings of the International Conference on Machine Learning (ICML’13)
  43. Mens Kim, Recommendation Systems in Software Engineering
  44. Mou Lili, 2016, Proceedings of the Conference of Artificial Intelligence (AAAI’16)
  45. Movshovitz-Attias Dana, Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL’13)
  46. Movshovitz-Attias Dana, 2015, Cohen
  47. Murali Vijayaraghavan, 2018, Proceedings of the International Conference on Learning Representations (ICLR).
  48. Neelakantan Arvind, 2015, Proceedings of the International Conference on Learning Representations (ICLR’15)
  49. Nguyen Anh Tuan, Proceedings of the International Conference on Software Engineering (ICSE’15)
  50. Nguyen Anh Tuan, Proceedings of the International Symposium on Foundations of Software Engineering (FSE’13)
  51. Nguyen Anh T., Proceedings of the International Conference on Automated Software Engineering (ASE’15)
  52. Nguyen Trong Duc, Proceedings of the International Conference on Software Engineering (ICSE’16)
  53. Nguyen Trong Duc, Proceedings of the International Conference on Software Engineering (ICSE’17)
  54. Nguyen Tung Thanh, Proceedings of the Joint Meeting of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering (ESEC/FSE’13)
  55. Omar Cyrus, 2017, Jonathan Aldrich, and Matthew A. Hammer.
  56. Parisotto Emilio, 2017, Proceedings of the International Conference on Learning Representations (ICLR’17)
  57. Pham Hung Viet, 2016, Proceedings of the International Conference on Software Engineering (ICSE’16)
  58. Piech Chris, Proceedings of the International Conference on Machine Learning (ICML’15)
  59. 10.5555/1667583.1667599 · DOI
  60. Reed Scott, 2016, Proceedings of the International Conference on Learning Representations (ICLR’16)
  61. Riedel Sebastian, 2017, Proceedings of the International Conference on Machine Learning (ICML’17)
  62. Rocktäschel Tim, 2017, Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS’17)
  63. 10.18653/v1/P16-1162 · DOI · Paper page
  64. Sutskever Ilya, Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS’14)
  65. Xin Wang Chang Liu Richard Shin Joseph E. Gonzalez and Dawn Song. 2016. Neural Code Completion. Retrieved from https://openreview.net/pdf?id=rJbPBt9lg. Xin Wang Chang Liu Richard Shin Joseph E. Gonzalez and Dawn Song. 2016. Neural Code Completion. Retrieved from https://openreview.net/pdf?id=rJbPBt9lg.
  66. 10.5555/2820518.2820559 · DOI
  67. Witten Ian H., 2016, Pal
  68. Zheng Alice X., 2003, Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS’03)

Cited by

Sorted by citations

Items 101–161 of 161

Related

See all

Metadata

Authors
Allamanis, M., Barr, E. T., Devanbu, P., & Sutton, C.
Affiliations
  • Microsoft Research (United Kingdom) (GB)
  • University College London (GB)
  • University of California, Davis, California, USA
  • The Alan Turing Institute (GB)
ORCID
Venue
ACM Computing Surveys
Publisher
Association for Computing Machinery (ACM)
Published
2018-07-31
Volume
51
Issue
4
Pages
1-37
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
DOI
10.1145/3212695
OpenAlex
W2754629507
Cited by
150
References
186
Citations by year
2026: 27 · 2025: 11 · 2024: 7 · 2023: 33 · 2022: 19 · 2021: 19 · 2020: 10 · 2019: 15 · 2018: 8
FWCI
20.98
Type
journal article
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • ACM Computing Surveys (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • UCL Discovery (University College London) (submittedVersion)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • Engineering and Physical Sciences Research Council (EP/K024043/1, EP/P005659/1, and EP/P005314/1)
  • National Science Foundation (1414172)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Logic, programming, and type systems
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
probabilistic programming languages, naturalness, software engineering applications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review