Studying the difference between natural and programming language corpora
Open accessThe claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The current methods are primarily tailored for natural language processing tasks, and it is not yet clear whether these techniques can be effectively applied within the realm of software engineering, since the source code fundamentally differs from the natural language.
“The current methods are primarily tailored for natural language processing tasks, and it is not yet clear whether these techniques can be effectively applied within the realm of software engineering, since the source code fundamentally differs from the natural language [30].”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [30]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb00917.x · Cited by 49,574 · see citing works
ggplot2: Elegant Graphics for Data Analysis
Wickham, H. (2009) · doi:10.1007/978-0-387-98141-3 · Cited by 14,447 · see citing works
Recurrent neural network based language model
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., & Khudanpur, S. (2010) · Interspeech 2010 · doi:10.21437/interspeech.2010-343 · Cited by 5,390 · see citing works
Prediction and Entropy of Printed English
Shannon, C. E. (1951) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1951.tb01366.x · Cited by 2,686 · see citing works
Expectation-based syntactic comprehension
Lévy, R. (2007) · Cognition · doi:10.1016/j.cognition.2007.05.006 · Cited by 2,220 · see citing works
LSTM neural networks for language modeling
Sundermeyer, M., Schlüter, R., & Ney, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-65 · Cited by 1,963 · see citing works
Knuth, D. (1984) · Computer/law journal · doi:10.1093/comjnl/27.2.97 · Cited by 1,863 · see citing works
A Brief History of Generative Models for Power Law and Lognormal Distributions
Mitzenmacher, M. (2004) · Internet Mathematics · doi:10.1080/15427951.2004.10129088 · Cited by 1,809 · see citing works
Improved backing-off for M-gram language modeling
Kneser, R., & Ney, H. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1995.479394 · Cited by 1,471 · see citing works
Programs, life cycles, and laws of software evolution
Lehman, M. M. (1980) · Proceedings of the IEEE · doi:10.1109/proc.1980.11805 · Cited by 1,181 · see citing works
The Stanford typed dependencies representation
de Marneffe, M.-C., & Manning, C. D. (2008) · doi:10.3115/1608858.1608859 · Cited by 926 · see citing works
Zipf’s word frequency law in natural language: A critical review and future directions
Piantadosi, S. T. (2014) · Psychonomic Bulletin & Review · doi:10.3758/s13423-014-0585-6 · Cited by 894 · see citing works
A Lego System for Conditional Inference
Hothorn, T., Hornik, K., van de Wiel, M. A., & Zeileis, A. (2006) · The American Statistician · doi:10.1198/000313006x118430 · Cited by 894 · see citing works
NLTK: The Natural Language Toolkit
Bird, S. (2006) · Proceedings of the COLING/ACL on Interactive presentation sessions - · doi:10.3115/1225403.1225421 · Cited by 841 · see citing works
Learning Accurate, Compact, and Interpretable Tree Annotation
Petrov, S., Barrett, L., Thibaux, R., & Klein, D. (2006) · Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the ACL - ACL '06 · doi:10.3115/1220175.1220230 · Cited by 793 · see citing works
Conklin, K., & Schmitt, N. (2007) · Applied Linguistics · doi:10.1093/applin/amm022 · Cited by 654 · see citing works
The communicative function of ambiguity in language
Piantadosi, S. T., Tily, H., & Gibson, E. (2012) · Cognition · doi:10.1016/j.cognition.2011.10.004 · Cited by 628 · see citing works
Laws of software evolution revisited
Lehman, M. M. (1996) · Lecture notes in computer science · doi:10.1007/bfb0017737 · Cited by 578 · see citing works
Data from eye-tracking corpora as evidence for theories of syntactic processing complexity
Demberg, V., & Keller, F. (2008) · Cognition · doi:10.1016/j.cognition.2008.07.008 · Cited by 575 · see citing works
Globally Normalized Transition-Based Neural Networks
Andor, D., Alberti, C., Weiss, D., Severyn, A., Presta, A., Ganchev, K., Petrov, S., & Collins, M. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1231 · Cited by 494 · see citing works
Influence of social and technical factors for evaluating contribution in GitHub
Tsay, J., Dabbish, L., & Herbsleb, J. D. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568315 · Cited by 447 · see citing works
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425 · see citing works
Self-citationLearning Cross-Modal Embeddings for Cooking Recipes and Food Images
Salvador, A., Hynes, N., Aytar, Y., Marín, J., Ofli, F., Weber, I. G., & Torralba, A. B. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.327 · Cited by 416 · see citing works
Mining source code repositories at massive scale using language modeling
Allamanis, M., & Sutton, C. (2013) · IEEE Working Conference on Mining Software Repositories · doi:10.1109/msr.2013.6624029 · Cited by 393 · see citing works
Danet, B. (1980) · Law & Society Review · doi:10.2307/3053192 · Cited by 376 · see citing works
Formulaic Language in Learner Corpora
Paquot, M., & Granger, S. (2012) · Annual Review of Applied Linguistics · doi:10.1017/s0267190512000098 · Cited by 346 · see citing works
The Information Conveyed by Words in Sentences
Hale, J. (2003) · Journal of Psycholinguistic Research · doi:10.1023/a:1022492123056 · Cited by 284 · see citing works
Two Regimes in the Frequency of Words and the Origins of Complex Lexicons: Zipf’s Law Revisited ∗
Ferrer‐i‐Cancho, R., & Solé, R. V. (2001) · Journal of Quantitative Linguistics · doi:10.1076/jqul.8.3.165.4101 · Cited by 242 · see citing works
Tu, Z., Su, Z., & Devanbu, P. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635875 · Cited by 238 · see citing works
Power-Laws in a Large Object-Oriented Software System
Concas, G., Marchesi, M., Pinna, S., et al. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.1019 · Cited by 228 · see citing works
Toward Deep Learning Software Repositories
White, M., Vendome, C., Linares‐Vásquez, M., & Poshyvanyk, D. (2015) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2015.38 · Cited by 222 · see citing works
Understanding understanding source code with functional magnetic resonance imaging
Siegmund, J., Kästner, C., Apel, S., Parnin, C., Bethmann, A., Leich, T., Saake, G., & Brechmann, A. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568252 · Cited by 219 · see citing works
Schmitt, N., & Carter, R. (2004) · Language learning and language teaching · doi:10.1075/lllt.9.02sch · Cited by 216 · see citing works
GHTorrent: Github's data from a firehose
Gousios, G., & Spinellis, D. D. (2012) · doi:10.1109/msr.2012.6224294 · Cited by 210 · see citing works
Weintrop, D., & Wilensky, U. (2015) · Proceedings of the eleventh annual International Conference on International Computing Education Research · doi:10.1145/2787622.2787721 · Cited by 198 · see citing works
Languages cool as they expand: Allometric scaling and the decreasing need for new words
Petersen, A. M., Tenenbaum, J. N., Havlin, S., Stanley, H. E., & Perc, M. (2012) · Scientific Reports · doi:10.1038/srep00943 · Cited by 192 · see citing works
Eye Movements in Code Reading: Relaxing the Linear Order
Busjahn, T., Bednarik, R., Begel, A., Crosby, M. E., Paterson, J. H., Schulte, C., Sharif, B., & Tamm, S. (2015) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2015.36 · Cited by 182 · see citing works
(2014) · doi:10.1145/2635868.2635901 · Cited by 159 · see citing works
The role of refactorings in API evolution
Dig, D., & Johnson, R. E. (2005) · Proceedings/Proceedings - Conference on Software Maintenance · doi:10.1109/icsm.2005.90 · Cited by 157 · see citing works
An empirical investigation into the role of API-level refactorings during software evolution
Kim, M., Cai, D., & Kim, S. (2011) · Proceedings/Proceedings - International Conference on Software Engineering · doi:10.1145/1985793.1985815 · Cited by 141 · see citing works
Uncertainty Reduction as a Measure of Cognitive Load in Sentence Comprehension
Frank, S. L. (2013) · Topics in Cognitive Science · doi:10.1111/tops.12025 · Cited by 139 · see citing works
Cost, benefits and quality of software development documentation: A systematic mapping
Zhi, J., Garousi-Yusifoğlu, V., Sun, B., Garousi, G., Shahnewaz, S., & Ruhe, G. (2015) · Journal of Systems and Software · doi:10.1016/j.jss.2014.09.042 · Cited by 112 · see citing works
Automatically assessing code understandability: How far are we?
Scalabrino, S., Bavota, G., Vendome, C., Linares‐Vásquez, M., Poshyvanyk, D., & Oliveto, R. (2017) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115654 · Cited by 101 · see citing works
Chomsky, N. (2002) · Cambridge University Press eBooks · doi:10.1017/cbo9780511613876.005 · Cited by 90 · see citing works
Software process data quality and characteristics
Bachmann, A., & Bernstein, A. (2009) · doi:10.1145/1595808.1595830 · Cited by 80 · see citing works
Morphology and the web of grammar : essays in memory of Steven G. Lapointe
Orgun, C. O., Sells, P., & Lapointe, S. G. (2005) · Cited by 75 · see citing works
How programmers read regular code: a controlled experiment using eye tracking
Jbara, A., & Feitelson, D. G. (2016) · Empirical Software Engineering · doi:10.1007/s10664-016-9477-x · Cited by 68 · see citing works
Understanding misunderstandings in source code
Gopstein, D., Iannacone, J., Yu, Y., DeLong, L. A., Zhuang, Y., Yeh, M. K. ‐., & Cappos, J. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106264 · Cited by 65 · see citing works
Improving After-the-Fact Tracing and Mapping: Supporting Software Quality Predictions
Hayes, J. H., Dekhtyar, A., & Sundaram, S. K. (2005) · IEEE Software · doi:10.1109/ms.2005.156 · Cited by 45 · see citing works
Will They Like This? Evaluating Code Contributions with Language Models
Hellendoorn, V. J., Dévanbu, P., & Bacchelli, A. (2015) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2015.22 · Cited by 45 · see citing works
Stochastic Optimization of Program Obfuscation
Liu, H., Sun, C. P., Su, Z., Jiang, Y., Gu, M., & Sun, J. (2017) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2017.28 · Cited by 35 · see citing works
Prevalence of confusing code in software projects
Gopstein, D., Zhou, H. H., Frankl, P. G., et al. (2018) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3196398.3196432 · Cited by 34 · see citing works
"Automatically assessing code understandability" reanalyzed
Trockman, A., Cates, K., Mozina, M., et al. (2018) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3196398.3196441 · Cited by 33 · see citing works
The Quorum Programming Language (Abstract Only)
Stefik, A., & Ladner, R. E. (2017) · doi:10.1145/3017680.3022377 · Cited by 31 · see citing works
Santos, E. A., & Hindle, A. (2016) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/2901739.2903493 · Cited by 29 · see citing works
Perceived language complexity in GitHub issue discussions and their effect on issue resolution
Kavaler, D., Sirovica, S., Hellendoorn, V. J., Aranovich, R., & Filkov, V. (2017) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2017.8115620 · Cited by 29 · see citing works
Social Factors in Language Change
Bright, W. (2017) · doi:10.1002/9781405166256.ch5 · Cited by 25 · see citing works
Defining the undefinedness of C
Hathhorn, C., Ellison, C., & Roşu, G. (2015) · ACM SIGPLAN Notices · doi:10.1145/2813885.2737979 · Cited by 25 · see citing works
Special Language and General Language: Linguistic and Didactic Aspects
Varantola, K. (1986) · Cited by 24 · see citing works
Exploring Regularity in Source Code: Software Science and Zipf's Law
Zhang, H. (2008) · Proceedings - Working Conference on Reverse Engineering · doi:10.1109/wcre.2008.37 · Cited by 22 · see citing works
Baselines in requirements tracing
Sundaram, S. K., Hayes, J. H., & Dekhtyar, A. (2005) · doi:10.1145/1083165.1083169 · Cited by 20 · see citing works
An empirical exploration of regularities in open-source software lexicons
Pierret, D., & Poshyvanyk, D. (2009) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2009.5090047 · Cited by 14 · see citing works
Social dimensions of language change
Michael, L. (2015) · Routledge eBooks · doi:10.4324/9781315794013.ch22 · Cited by 10 · see citing works
Flagged first, then by citations
Research works printed but not linked: 16
- Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M, et al (2016) Tensorflow: A system for large-scale machine learning. In: OSDI, vol 16, pp 265-283
- Allamanis M, Barr ET, Devanbu P, Sutton C (2017) A survey of machine learning for big code and naturalness (to appear, acm computing surveys). arXiv preprint arXiv:170906182
- Baxter G, Frean M, Noble J, Rickerby M, Smith H, Visser M, Melton H, Tempero E (2006) Understanding the shape of java software. In: ACM Sigplan Notices, ACM, vol 41, pp 397-412
- Bradley DC (1978) Computational distinctions of vocabulary type. PhD thesis, Mas- sachusetts Institute of Technology
- Chelba C, Mikolov T, Schuster M, Ge Q, Brants T, Koehn P (2013) One billion word bench- mark for measuring progress in statistical language modeling. CoRR abs/1312.3005, URL http://arxiv.org/abs/1312.3005 · arXiv
- De Marneffe MC, MacCartney B, Manning CD, et al (2006) Generating typed dependency parses from phrase structure parses. In: Proceedings of LREC, Genoa Italy, vol 6, pp 449-454
- Gerlach M, Altmann EG (2013) Stochastic model for the vocabulary growth in natural lan- guages. Physical Review X 3(2):021006
- Ginter F, Hajič J, Luotolahti J, Straka M, Zeman D (2017) CoNLL 2017 shared task -auto- matically annotated raw texts and word embeddings. URL http://hdl.handle.net/ 11234/1-1989, LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics ( ÚFAL), Faculty of Mathematics and Physics, Charles University
- Heafield K (2011) Kenlm: Faster and smaller language model queries. In: Proceedings of the Sixth Workshop on Statistical Machine Translation, Association for Computational Linguistics, pp 187-197
- Hellendoorn VJ, Devanbu P (2017) Are deep neural networks the best choice for modeling source code? In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ser. ESEC/FSE, pp 763-773
- Khanh Dam H, Tran T, Pham T (2016) A deep language model for software code. arXiv preprint arXiv:160802715
- Louridas P, Spinellis D, Vlachos V (2008) Power laws in software. ACM Transactions on Software Engineering and Methodology (TOSEM) 18(1):2
- Manning CD, Schütze H (1999) Foundations of statistical natural language processing. MIT press
- Marcus MP, Marcinkiewicz MA, Santorini B (1993) Building a large annotated corpus of english: The penn treebank. Comput Linguist 19(2):313-330, URL http://dl.acm. org/citation.cfm?id=972470.972475
- Vinyals O, Kaiser Ł, Koo T, Petrov S, Sutskever I, Hinton G (2015) Grammar as a foreign language. In: Advances in Neural Information Processing Systems, pp 2773-2781
- Zipf G (1949) Human behavior and the principle of least effort. Addison-Wesley, Cambody Mus Am Arch and Ethnol(Harvard Univ), Papers 19:1-125
Cited by
Sorted by citations
A 2030 Roadmap for Software Engineering
Pezzè, M., Abrahão, S., Penzenstadler, B., Poshyvanyk, D., Roychoudhury, A., & Yue, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3731559 · Cited by 13 · see citing works
- Authors
- Casalnuovo, C., Sagae, K., & Devanbu, P.
- Affiliations
- University of California, Davis (US)
- Corresponding author
- Casalnuovo, Casey
- Venue
- Empirical Software Engineering
- Published
- 2019-01-11
- Volume
- 24
- Issue
- 4
- Pages
- 1823-1868
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- OpenAlex
- W2807585453
- Cited by
- 33
- References
- 65
- Citations by year
- 2026: 4 · 2025: 8 · 2024: 3 · 2023: 4 · 2022: 4 · 2021: 6 · 2020: 2 · 2019: 2
- FWCI
- 4.36
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Empirical Software Engineering (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1414172)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, programming, and type systems
- Keywords
- source code, natural language corpora, programming language syntax, technical jargon, second language writing, code completion, code readability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
