Error Message Readability and Novice Debugging Performance
Open accessAbstract
It is well known that programming error messages can be notoriously difficult for novices to understand, hampering progress and leading to frustration. In response, researchers have explored various approaches for enhancing such messages, yet results from this active strand of research are currently mixed. Direct comparisons of results between studies is challenging as these typically investigate different kinds of message enhancements and report results using different metrics. In addition, many prior studies have involved code writing tasks. In such cases, not all students encounter the same errors and messages, and it is difficult to isolate the time spent interpreting messages and resolving errors from the time spent writing code. In this research, we explore the effects of presenting novices with compiler error messages designed using the most recent collection of published guidelines - specifically, more easily readable, short, positive messages containing resolution hints. To accurately determine the time and effort required to read and respond to the messages, we utilise a debugging task where all students are presented the same code and therefore encounter the same errors. We present results of a randomised controlled experiment (n > 700) which shows that, compared to standard error messages, the messages we tested resulted in significantly shorter debugging times and higher self-reported scores of message usefulness for students in the very early stages of learning a new language.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Increasing the readability of programming error messages improves student understanding of error messages and the number of successful code corrections.
“This has been shown to improve student understanding of error messages and the number of successful code corrections [16].”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [16]
Although some studies have shown positive effects of rewording messages for novices, in general the evidence for the effectiveness of enhanced compiler error messages is not overwhelming.
“However, although some studies have shown positive effects of rewording messages for novices [9,16], in general the evidence for the effectiveness of enhanced compiler error messages is not overwhelming [7].”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [16]
Traditional error messages are often cryptic and hard to understand for novice programmers.
“traditional error messages are often cryptic and hard to understand for novice programmers [16,17].”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [16]
Cites
Why don't software developers use static analysis tools to find bugs?
Johnson, B., Song, Y., Murphy-Hill, E., & Bowdidge, R. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606613 · Cited by 568 · cited 2× · see citing works
“Inadequate explanation of the problem causing an error is a key reason why developers report finding messages unhelpful [26].”
BackgroundThe BlueJ System and its Pedagogy
Kölling, M., Quig, B., Patterson, A., & Rosenberg, J. (2003) · Computer Science Education · doi:10.1076/csed.13.4.249.17496 · Cited by 484 · see citing works
What would other programmers do
Hartmann, B., MacDougall, D., Brandt, J., & Klemmer, S. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753478 · Cited by 280 · see citing works
Five-Point Likert Items: t test versus Mann-Whitney-Wilcoxon (Addendum added October 2012)
de Winter, J. F. C., & Dodou, D. (2010) · University of Massachusetts Amherst · doi:10.7275/bj1p-ts64 · Cited by 231 · see citing works
Becker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372508 · Cited by 199 · see citing works
Self-citationBrown, N. C. C., Kölling, M., McCall, D., & Utting, I. (2014) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2538862.2538924 · Cited by 179 · see citing works
Enhancing syntax error messages appears ineffectual
Denny, P. C., Luxton-Reilly, A., & Carpenter, D. (2014) · doi:10.1145/2591708.2591748 · Cited by 148 · see citing works
Self-citationAll syntax errors are not equal
Denny, P., Luxton-Reilly, A., & Tempero, E. (2012) · Proceedings of the 17th ACM annual conference on Innovation and technology in computer science education · doi:10.1145/2325296.2325318 · Cited by 144 · see citing works
Self-citationUnderstanding the syntax barrier for novices
Denny, P., Luxton-Reilly, A., Tempero, E., & Hendrickx, J. (2011) · Proceedings of the 16th annual joint conference on Innovation and technology in computer science education · doi:10.1145/1999747.1999807 · Cited by 133 · see citing works
Self-citationAn Effective Approach to Enhancing Compiler Error Messages
Becker, B. A. (2016) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2839509.2844584 · Cited by 125 · see citing works
Self-citation50 Years of CS1 at SIGCSE: A Review of the Evolution of Introductory Programming Education Research
Becker, B. A., & Quille, K. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287432 · Cited by 107 · see citing works
Self-citationDo Developers Read Compiler Error Messages?
Barik, T., Smith, J., Lubick, K., Holmes, E., Feng, J., Murphy-Hill, E., & Parnin, C. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.59 · Cited by 86 · cited 3× · see citing works
“Even more striking, the difficulty of reading error messages was shown to be comparable to that of reading source code itself [3].”
BackgroundResultCompiler Construction: An Advanced Course
Bauer, F. L., Remer, F. L. D., Griffiths, M., et al. (1974) · Medical Entomology and Zoology · Cited by 85 · see citing works
The Effects of Enhanced Compiler Error Messages on a Syntax Error Debugging Test
Becker, B. A., Goslin, K., & Glanville, G. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159461 · Cited by 62 · see citing works
Self-citationFix the First, Ignore the Rest: Dealing with Multiple Compiler Error Messages
Becker, B. A., Murray, C., Tao, T., Song, C., McCartney, R., & Sanders, K. (2018) · Proceedings of the 49th ACM Technical Symposium on Computer Science Education · doi:10.1145/3159450.3159453 · Cited by 61 · see citing works
Self-citationIDE-Based Learning Analytics for Computing Education
Hundhausen, C. D., Olivares, D. M., & Carter, A. S. (2017) · ACM Transactions on Computing Education · doi:10.1145/3105759 · Cited by 60 · see citing works
The Error Behind The Message: Finding the Cause of Error Messages in Python
Kohn, T. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287381 · Cited by 41 · see citing works
How should compilers explain problems to developers?
Barik, T., Ford, D., Murphy-Hill, E. R., & Parnin, C. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236040 · Cited by 40 · see citing works
The Effects of Compilation Mechanisms and Error Message Presentation on Novice Programmer Behavior
Karvelas, I., Li, A. D. R., & Becker, B. A. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366882 · Cited by 33 · see citing works
Self-citationHow a computer should talk to people
Dean, M. (1982) · IBM Systems Journal · doi:10.1147/sj.214.0424 · Cited by 25 · see citing works
An Exploration of the Effects of Enhanced Compiler Error Messages for Computer Programming Novices
Becker, B. A. (2015) · Arrow - TU Dublin (Technological University Dublin) · doi:10.13140/rg.2.2.26637.13288 · Cited by 22 · see citing works
Self-citationStudent difficulties with pointer concepts in C
Craig, M., & Petersen, A. (2016) · Australasian Computer Science Week Multiconference (ACSW) · doi:10.1145/2843043.2843348 · Cited by 21 · see citing works
Kohn, T. (2017) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-a-010871088 · Cited by 15 · see citing works
The Impact of "Cosmetic" Changes on the Usability of Error Messages
Dong, T., & Khandwala, K. (2019) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3312978 · Cited by 13 · see citing works
Buxton, A., & Trenner, L. (1987) · Journal of Information Science · doi:10.1177/016555158701300403 · Cited by 6 · see citing works
A Mixed-Methods Study of Novice Programmer Interaction with Python Error Messages
D'souza, R., Bhayana, M., Ahmadzadeh, M., & Harrington, B. (2019) · Western Canadian Conference on Computing Education · doi:10.1145/3314994.3325090 · Cited by 6 · see citing works
Rules for the Design of End User Languages
Engelmann, U., & Meinzer, H. (1985) · Lecture notes in medical informatics · doi:10.1007/978-3-642-93295-3_48 · Cited by 3 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Automated Grading and Feedback Tools for Programming Education: A Systematic Review
Messer, M., Brown, N. C. C., Kölling, M., & Shi, M. (2024) · ACM Transactions on Computing Education · doi:10.1145/3636515 · Cited by 262 · see citing works
My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E. A., Prather, J., & Becker, B. A. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576134 · Cited by 186 · see citing works
Self-citationOn Designing Programming Error Messages for Novices: Readability and its Constituent Factors
Denny, P., Prather, J., Becker, B. A., Mooney, C., Homer, J., Albrecht, Z. C., & Powell, G. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445696 · Cited by 60 · see citing works
Self-citationA Comparison of Immediate and Scheduled Feedback in Introductory Programming Projects
Leinonen, J., Denny, P., & Whalley, J. (2022) · Proceedings of the 53rd ACM Technical Symposium on Computer Science Education · doi:10.1145/3478431.3499372 · Cited by 32 · see citing works
Self-citationDecoding Debugging Instruction: A Systematic Literature Review of Debugging Interventions
Yang, S., Baird, M., O’Rourke, E., Brennan, K., & Schneider, B. (2024) · ACM Transactions on Computing Education · doi:10.1145/3690652 · Cited by 29 · see citing works
Promoting Early Engagement with Programming Assignments Using Scheduled Automated Feedback
Denny, P., Whalley, J., & Leinonen, J. (2021) · Proceedings of the 23rd Australasian Computing Education Conference · doi:10.1145/3441636.3442309 · Cited by 24 · see citing works
Self-citationNovice Reflections During the Transition to a New Programming Language
Denny, P., Becker, B. A., Bosch, N., Prather, J., Reeves, B., & Whalley, J. (2022) · Proceedings of the 53rd ACM Technical Symposium on Computer Science Education · doi:10.1145/3478431.3499314 · Cited by 11 · see citing works
Self-citation
- Authors
- Denny, P., Prather, J., & Becker, B. A.
- Affiliations
- University of Auckland (NZ)
- Abilene Christian University (US)
- University College Dublin (IE)
- Venue
- Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education
- Publisher
- ACM
- Published
- 2020-06-15
- Pages
- 480-486
- OpenAlex
- W3033595121
- Cited by
- 50
- References
- 27
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 14 · 2023: 13 · 2022: 9 · 2021: 4 · 2020: 1
- FWCI
- 7.84
- Type
- conference paper
- Language
- en
- Open access
- green
- hdl.handle.net
- Licence
- Locations
- ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) (publishedVersion)
- Research Repository UCD (University College Dublin) (publishedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- novice programmers, compiler error messages, programming education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
