Hard-to-answer questions about code
The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
To build new tools and programming languages that make it easier for professional software developers to create, debug, and understand code, it is helpful to better understand the questions that developers ask during coding activities. We surveyed professional software developers and asked them to list hard-to-answer questions that they had recently asked about code. 179 respondents reported 371 questions. We then clustered these questions into 21 categories and 94 distinct questions. The most frequently reported categories dealt with intent and rationale -- what does this code do, what is it intended to do, and why was it done this way? Many questions described very specific situations -- e.g., what does the code do when an error occurs, how to refactor without breaking callers, or the implications of a specific change on security. These questions revealed opportunities for both existing research tools to help developers and for developing new languages and tools that make answering these questions easier.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
(1994) · Cited by 21,711
DART: directed automated random testing
Godefroid, P., Klarlund, N., & Sen, K. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065036 · Cited by 2,067
CCFinder: a multilinguistic token-based code clone detection system for large scale source code
Kamiya, T., Kusumoto, S., & Inoue, K. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1019480 · Cited by 1,711
DART: directed automated random testing
Godefroid, P., Klarlund, N., & Sen, K. (2005) · Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation · doi:10.1145/1065010.1065036 · Cited by 1,570
Anvik, J., Hiew, L., & Murphy, G. C. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134336 · Cited by 924
An overview of JML tools and applications
Burdy, L., Cheon, Y., Cok, D. R., Ernst, M. D., Kiniry, J. R., Leavens, G. T., Leino, K. R. M., & Poll, E. (2005) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-004-0167-4 · Cited by 714
Maintaining mental models: a study of developer work habits
LaToza, T. D., Venolia, G., & DeLine, R. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134355 · Cited by 628
Self-citationTypestate: A programming language concept for enhancing software reliability
Strom, R. E., & Yemini, S. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312929 · Cited by 626
Information Needs in Collocated Software Development Teams
Ko, A. J., DeLine, R., & Venolia, G. (2007) · 29th International Conference on Software Engineering (ICSE'07) · doi:10.1109/icse.2007.45 · Cited by 499 · cited 8×
“One study found 21 questions about interactions with code, other artifacts, and teammates [15].”
BackgroundInfluentialException handling: issues and a proposed notation
Goodenough, J. B. (1975) · Communications of the ACM · doi:10.1145/361227.361230 · Cited by 496
Expertise Browser: a quantitative approach to identifying expertise
Mockus, A., & Herbsleb, J. D. (2003) · Proceedings of the 24th International Conference on Software Engineering. ICSE 2002 · doi:10.1109/icse.2002.1007994 · Cited by 383
Design Rationale: Concepts, Techniques, and Use
(2020) · CRC Press eBooks · doi:10.1201/9781003064053 · Cited by 371
Using dependency models to manage complex software architecture
Sangal, N., Jordan, E., Sinha, V., & Jackson, D. (2005) · Proceedings of the 20th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications · doi:10.1145/1094811.1094824 · Cited by 369
Cognitive processes in program comprehension
Letovsky, S. (1987) · Journal of Systems and Software · doi:10.1016/0164-1212(87)90032-x · Cited by 327
Asking and Answering Questions during a Programming Change Task
Sillito, J., Murphy, G. C., & De Volder, K. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.26 · Cited by 319
The structure and value of modularity in software design
Sullivan, K. J., Griswold, W. G., Cai, Y., & Hallen, B. (2001) · Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineering · doi:10.1145/503209.503224 · Cited by 299
Debugging reinvented: asking and answering why and why not questions about program behavior
Ko, A. J., & Myers, B. A. (2008) · Proceedings of the 30th international conference on Software engineering · doi:10.1145/1368088.1368130 · Cited by 296
Self-citationSridharan, M., Fink, S. J., & Bodik, R. (2007) · Proceedings of the 28th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/1250734.1250748 · Cited by 243
Design rationale: Concepts, techniques, and use
Dillon, A. P. (1997) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199708)48:8<718::aid-asi5>3.0.co;2-v · Cited by 227
Types for safe locking: Static race detection for Java
Abadi, M., Flanagan, C., & Freund, S. N. (2006) · ACM Transactions on Programming Languages and Systems · doi:10.1145/1119479.1119480 · Cited by 222
An overview of JML tools and applications1 1www.jmlspecs.org
Burdy, L., Cheon, Y., Cok, D., Ernst, M. D., Kiniry, J., Leavens, G. T., Rustan, K., Leino, M., & Poll, E. (2003) · Electronic Notes in Theoretical Computer Science · doi:10.1016/s1571-0661(04)80810-7 · Cited by 206
Expertise browser: a quantitative approach to identifying expertise
Mockus, A., & Herbsleb, J. D. (2002) · Proceedings of the 24th international conference on Software engineering - ICSE '02 · doi:10.1145/581339.581401 · Cited by 199
Universes: Lightweight Ownership for JML.
Dietl, W., & Müller, P. (2005) · The Journal of Object Technology · doi:10.5381/jot.2005.4.8.a1 · Cited by 176
Using information fragments to answer the questions developers ask
Fritz, T., & Murphy, G. C. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 · doi:10.1145/1806799.1806828 · Cited by 163
Developers ask reachability questions
LaToza, T. D., & Myers, B. A. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 · doi:10.1145/1806799.1806829 · Cited by 145
Self-citationAssieme: finding and leveraging implicit references in a web search interface for programmers
Hoffmann, R., Fogarty, J., & Weld, D. S. (2007) · Proceedings of the 20th annual ACM symposium on User interface software and technology · doi:10.1145/1294211.1294216 · Cited by 130
Program comprehension as fact finding
LaToza, T. D., Garlan, D., Herbsleb, J. D., & Myers, B. A. (2007) · Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1287624.1287675 · Cited by 105
Self-citationA study on the current state of the art in tool-supported UML-based static reverse engineering
Kollmann, R., Selonen, P., Stroulia, E., Systä, T., & Zündorf, A. (2003) · Working Conference on Reverse Engineering (WCRE) · doi:10.1109/wcre.2002.1173061 · Cited by 104
The Journal of Object Technology
(2017) · The Journal of Object Technology · doi:10.5381/jot · Cited by 99
Proving that programs eventually do something good
Cook, B., Gotsman, A., Podelski, A., Rybalchenko, A., & Vardi, M. Y. (2007) · ACM SIGPLAN Notices · doi:10.1145/1190215.1190257 · Cited by 88
Proving that programs eventually do something good
Cook, B., Gotsman, A., Podelski, A., Rybalchenko, A., & Vardi, M. Y. (2007) · ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) · doi:10.1145/1190216.1190257 · Cited by 71
Design rationale concepts, techniques, and use
Moran, T. P. (1996) · Cited by 58
Deep intellisense: a tool for rehydrating evaporated information
Holmes, R., & Begel, A. (2008) · Proceedings of the 2008 international working conference on Mining software repositories · doi:10.1145/1370750.1370755 · Cited by 54
Erickson, C. (2003) · Linux journal · Cited by 6
The code of many colors: semi-automated reasoning about multi-thread policy for java
Scherlis, W. L., & Sutherland, D. F. (2008) · Cited by 3
Cited by
How do programmers ask and answer questions on the web? (NIER track)
Treude, C., Barzilay, O., & Storey, M.-A. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985907 · Cited by 405
“LaToza and Myers [7] found that the most difficult questions from a developer’s perspective dealt with intent and rationale.”
BackgroundLearning to rank relevant files for bug reports using domain knowledge
Ye, X., Bunescu, R., & Liu, C. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635874 · Cited by 290
“A developer who is assigned a bug report usually needs to reproduce the abnormal behavior [22] and perform code reviews [2] in order to (cid:12)nd the cause.”
BackgroundCategorizing the Content of GitHub README Files
Prana, G. A. A., Treude, C., Thung, F., Atapattu, T., & Lo, D. (2019) · Empirical Software Engineering · doi:10.1007/s10664-018-9660-3 · Cited by 143
Exploring exploratory programming
Beth Kery, M., & Myers, B. A. (2017) · 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2017.8103446 · Cited by 123
“Whereas a software engineer might ask a code-centered question like: “In which version did this code appear?” [31], an exploratory programmer may frame their question around an experiment: “When I tried a RandomForest algorithm, how did that effect my model’s accuracy?” or “When I went through this loop 3 times, how d…”
MethodSelf-citationImproving bug localization with word embedding and enhanced convolutional neural networks
Xiao, Y., Keung, J., Bennin, K. E., & Mi, Q. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.08.002 · Cited by 128
“Thus, bug resolution activity usually requires considerable time and e ff ort [23] to improve software quality and ensure software integrity [5].”
Background
Metadata
- Authors
- LaToza, T. D., & Myers, B. A.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Evaluation and Usability of Programming Languages and Tools
- Publisher
- ACM
- Published
- 2010-10-17
- Pages
- 1-6
- OpenAlex
- W1987614102
- Semantic Scholar
- 89df5d321047ea11cace4f38cebb65a2af3cd625
- Cited by
- 141
- References
- 35
- Citations by year
- 2026: 5 · 2025: 10 · 2024: 13 · 2023: 5 · 2022: 13 · 2021: 7 · 2020: 6 · 2019: 13 · 2018: 9 · 2017: 11 · 2016: 10 · 2015: 12 · 2014: 8 · 2013: 10 · 2012: 6
- FWCI
- 4.27
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Evaluation and Usability of Programming Languages and Tools (publishedVersion)
- Funders
- National Science Foundation (CCF-0811610ITR CCR-0324770)
- Division of Computing and Communication Foundations (CCF-0811610ITR CCR-0324770)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- program comprehension, developer intent, debugging, refactoring, error handling, developer survey, programming language design, software development tools
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.