Hard-to-answer questions about code
The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
To build new tools and programming languages that make it easier for professional software developers to create, debug, and understand code, it is helpful to better understand the questions that developers ask during coding activities. We surveyed professional software developers and asked them to list hard-to-answer questions that they had recently asked about code. 179 respondents reported 371 questions. We then clustered these questions into 21 categories and 94 distinct questions. The most frequently reported categories dealt with intent and rationale -- what does this code do, what is it intended to do, and why was it done this way? Many questions described very specific situations -- e.g., what does the code do when an error occurs, how to refactor without breaking callers, or the implications of a specific change on security. These questions revealed opportunities for both existing research tools to help developers and for developing new languages and tools that make answering these questions easier.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
(1994) · Cited by 21,711
DART: directed automated random testing
Godefroid, P., Klarlund, N., & Sen, K. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065036 · Cited by 2,067
CCFinder: a multilinguistic token-based code clone detection system for large scale source code
Kamiya, T., Kusumoto, S., & Inoue, K. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2002.1019480 · Cited by 1,711
DART: directed automated random testing
Godefroid, P., Klarlund, N., & Sen, K. (2005) · Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation · doi:10.1145/1065010.1065036 · Cited by 1,570
Anvik, J., Hiew, L., & Murphy, G. C. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134336 · Cited by 924
An overview of JML tools and applications
Burdy, L., Cheon, Y., Cok, D. R., Ernst, M. D., Kiniry, J. R., Leavens, G. T., Leino, K. R. M., & Poll, E. (2005) · International Journal on Software Tools for Technology Transfer · doi:10.1007/s10009-004-0167-4 · Cited by 714
Maintaining mental models: a study of developer work habits
LaToza, T. D., Venolia, G., & DeLine, R. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134355 · Cited by 628
Self-citationTypestate: A programming language concept for enhancing software reliability
Strom, R. E., & Yemini, S. (1986) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1986.6312929 · Cited by 626
Information Needs in Collocated Software Development Teams
Ko, A. J., DeLine, R., & Venolia, G. (2007) · 29th International Conference on Software Engineering (ICSE'07) · doi:10.1109/icse.2007.45 · Cited by 499 · cited 8×
“One study found 21 questions about interactions with code, other artifacts, and teammates [15].”
BackgroundInfluentialException handling: issues and a proposed notation
Goodenough, J. B. (1975) · Communications of the ACM · doi:10.1145/361227.361230 · Cited by 496
Cited by
How do programmers ask and answer questions on the web? (NIER track)
Treude, C., Barzilay, O., & Storey, M.-A. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985907 · Cited by 405
Learning to rank relevant files for bug reports using domain knowledge
Ye, X., Bunescu, R., & Liu, C. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635874 · Cited by 290
Categorizing the Content of GitHub README Files
Prana, G. A. A., Treude, C., Thung, F., Atapattu, T., & Lo, D. (2018) · Empirical Software Engineering · doi:10.1007/s10664-018-9660-3 · Cited by 189
Exploring exploratory programming
Beth Kery, M., & Myers, B. A. (2017) · 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2017.8103446 · Cited by 123
“Whereas a software engineer might ask a code-centered question like: “In which version did this code appear?” [31], an exploratory programmer may frame their question around an experiment: “When I tried a RandomForest algorithm, how did that effect my model’s accuracy?” or “When I went through this loop 3 times, how d…”
MethodSelf-citationImproving bug localization with word embedding and enhanced convolutional neural networks
Xiao, Y., Keung, J., Bennin, K. E., & Mi, Q. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.08.002 · Cited by 127
Metadata
- Authors
- LaToza, T. D., & Myers, B. A.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Evaluation and Usability of Programming Languages and Tools
- Publisher
- ACM
- Published
- 2010-10-17
- Pages
- 1-6
- OpenAlex
- W1987614102
- Semantic Scholar
- 89df5d321047ea11cace4f38cebb65a2af3cd625
- Cited by
- 141
- References
- 35
- Citations by year
- 2026: 5 · 2025: 10 · 2024: 13 · 2023: 5 · 2022: 13 · 2021: 7 · 2020: 6 · 2019: 13 · 2018: 9 · 2017: 11 · 2016: 10 · 2015: 12 · 2014: 8 · 2013: 10 · 2012: 6
- FWCI
- 4.27
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Evaluation and Usability of Programming Languages and Tools (publishedVersion)
- Funders
- National Science Foundation (CCF-0811610ITR CCR-0324770)
- Division of Computing and Communication Foundations (CCF-0811610ITR CCR-0324770)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- program comprehension, developer intent, debugging, refactoring, error handling, developer survey, programming language design, software development tools
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Hard-to-answer questions about code. https://app.cochat.ai/papers/10.1145/1937117.1937125
- The paper
- LaToza, T. D., & Myers, B. A. (2010). Hard-to-answer questions about code. Evaluation and Usability of Programming Languages and Tools. https://doi.org/10.1145/1937117.1937125
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.