An Effective Approach to Enhancing Compiler Error Messages
Open accessThe DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Abstract
One of the many challenges novice programmers face from the time they write their first program is inadequate compiler error messages. These messages report details on errors the programmer has made and are the only feedback the programmer gets from the compiler. For students they play a particularly essential role as students often have little experience to draw upon, leaving compiler error messages as their primary guidance on error correction. However these messages are frequently inadequate, presenting a barrier to progress and are often a source of discouragement. We have designed and implemented an editor that provides enhanced compiler error messages and conducted a controlled empirical study with CS1 students learning Java. We find a reduced frequency of overall errors and errors per student. We also identify eight frequent compiler error messages for which enhancement has a statistically significant effect. Finally we find a reduced number of repeated errors. These findings indicate fewer students struggling with compiler error messages.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Empirical Investigation into Programming Language Syntax
Stefik, A., & Siebert, S. (2013) · ACM Transactions on Computing Education · doi:10.1145/2534973 · Cited by 295 · cited 1×
“Unfortunately they are frequently inadequate and pose a significant barrier to progress [21],”
Background
Cited by
Students’ Misconceptions and Other Difficulties in Introductory Programming: A Literature Review
Qian, Y., & Lehman, J. (2018) · ACM Transactions on Computing Education · doi:10.1145/3077618 · Cited by 529
Introductory programming: a systematic literature review
Luxton-Reilly, A., Simon, Albluwi, I., Becker, B. A., Giannakos, M., Kumar, A. N., Ott, L., Paterson, J., Scott, M. J., Sheard, J., et al. (2018) · Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3293881.3295779 · Cited by 485
Self-citationAutomated Grading and Feedback Tools for Programming Education: A Systematic Review
Messer, M., Brown, N. C. C., Kölling, M., & Shi, M. (2024) · ACM Transactions on Computing Education · doi:10.1145/3636515 · Cited by 262
“Songet al. [176] developed an alternative program repair-based feedback tool for OCaml, utilising multiple partially matching solutions and test cases to generate a fix for logical errors.”
BackgroundMethodUsing Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235
Self-citationBecker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372508 · Cited by 199
Self-citationMetacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools
Prather, J., Pettit, R., McMurry, K., Peters, A., Homer, J., & Cohen, M. (2018) · Proceedings of the 2018 ACM Conference on International Computing Education Research · doi:10.1145/3230977.3230981 · Cited by 129
Effective compiler error message enhancement for novice programming students
Becker, B. A., Glanville, G., Iwashima, R., McDonnell, C., Goslin, K., & Mooney, C. (2016) · Computer Science Education · doi:10.1080/08993408.2016.1225464 · Cited by 101
Self-citationOn Novices' Interaction with Compiler Error Messages: A Human Factors Approach
Prather, J., Pettit, R., McMurry, K. H., Peters, A., Homer, J., Simone, N., & Cohen, M. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106169 · Cited by 72
“One research tool often employed in evaluating changes made to CS1 classes is the think-aloud protocol where students are observed writing code and asked to verbalize their thoughts while doing so.”
BackgroundMethodInfluentialDo Enhanced Compiler Error Messages Help Students?: Results Inconclusive.
Pettit, R. S., Homer, J., & Gee, R. (2017) · Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education · doi:10.1145/3017680.3017768 · Cited by 66
Fix the First, Ignore the Rest: Dealing with Multiple Compiler Error Messages
Becker, B. A., Murray, C., Tao, T., Song, C., McCartney, R., & Sanders, K. (2018) · Proceedings of the 49th ACM Technical Symposium on Computer Science Education · doi:10.1145/3159450.3159453 · Cited by 61
“[4, 5, 7] used an editor designed to collect and enhance error messages generated by the standard JDK compiler.”
MethodSelf-citationOn Designing Programming Error Messages for Novices: Readability and its Constituent Factors
Denny, P., Prather, J., Becker, B. A., Mooney, C., Homer, J., Albrecht, Z. C., & Powell, G. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445696 · Cited by 60
Self-citationError Message Readability and Novice Debugging Performance
Denny, P., Prather, J., & Becker, B. A. (2020) · Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3341525.3387384 · Cited by 50
Self-citationStatic Analyses in Python Programming Courses
Liu, D., & Petersen, A. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287503 · Cited by 46
Qian, Y., & Lehman, J. D. (2019) · Sage Open · doi:10.1177/2158244019885136 · Cited by 42
The Error Behind The Message: Finding the Cause of Error Messages in Python
Kohn, T. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287381 · Cited by 41
Mailach, A., Gorgosch, D., Siegmund, N., & Siegmund, J. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10561-6 · Cited by 13
“…chatbottounderstandlongstacktracesanderrormessagesalignswithinsightsfromresearch targetingthecomprehensibilityofcompilererrormessages,whichshowedthatenhanced errormessageswithexplanationsinnaturallanguage,helpprogrammingbeginnerstoreduce theoverallnumberoferrorstheymake(Becker2016).”
Metadata
- Authors
- Becker, B. A.
- Venue
- Proceedings of the 47th ACM Technical Symposium on Computing Science Education
- Publisher
- ACM
- Published
- 2016-02-17
- Pages
- 126-131
- OpenAlex
- W2296095100
- Cited by
- 125
- References
- 1
- FWCI
- 9.36
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.