Compiler Error Messages Considered Unhelpful: The Landscape of Text-Based Programming Error Message Research
Abstract
Diagnostic messages generated by compilers and interpreters such as syntax error messages have been researched for over half of a century. Unfortunately, these messages which include error, warning, and run-time messages, present substantial difficulty and could be more effective, particularly for novices. Recent years have seen an increased number of papers in the area including studies on the effectiveness of these messages, improving or enhancing them, and their usefulness as a part of programming process data that can be used to predict student performance, track student progress, and tailor learning plans. Despite this increased interest, the long history of literature is quite scattered and has not been brought together in any digestible form.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Leinonen et al. suggest that LLMs could help students understand terse error messages which have traditionally been a source of difficulty for novice learners.
“Leinonen et al. suggest that LLMs could help students understand terse error messages [125] which have traditionally been a source of difficulty for novice learners [36].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [36]
For over six decades, researchers have attempted to improve programming error messages, and there is still a call for more work on the topic.
“For over six decades, researchers have attempted to improve these messages, and still there is a call for more work on the topic [7].”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [7]
Nearly sixty years later, programming error messages are still a barrier to progress for those learning to program.
“Nearly sixty years later, PEMs are still a barrier to progress for those learning to program [7]”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [7]
Although some studies have shown positive effects of rewording messages for novices, in general the evidence for the effectiveness of enhanced compiler error messages is not overwhelming.
“However, although some studies have shown positive effects of rewording messages for novices [9,16], in general the evidence for the effectiveness of enhanced compiler error messages is not overwhelming [7].”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [7]
Cites
Cognitive Load During Problem Solving: Effects on Learning
Sweller, J. (1988) · Cognitive Science · doi:10.1016/0364-0213(88)90023-7 · Cited by 11,048 · cited 1× · see citing works
“Cognitive Load Theory (CLT) states that humans have a finite ability to efficiently process input based on working memory (intrinsic) and situational/contextual (extrinsic) factors [195].”
BackgroundAn Empirical Investigation into Programming Language Syntax
Stefik, A., & Siebert, S. (2013) · ACM Transactions on Computing Education · doi:10.1145/2534973 · Cited by 295 · see citing works
Programming, Problem Solving, and Self-Awareness: Effects of Explicit Guidance
Loksa, D., Ko, A. J., Jernigan, W., Oleson, A., Mendez, C. J., & Burnett, M. M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858252 · Cited by 223 · see citing works
Do Developers Read Compiler Error Messages?
Barik, T., Smith, J., Lubick, K., Holmes, E., Feng, J., Murphy-Hill, E., & Parnin, C. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.59 · Cited by 86 · cited 2× · see citing works
“Rightfully so, students do not trust them [97] but importantly they do read them [12, 163].”
BackgroundDo Enhanced Compiler Error Messages Help Students?: Results Inconclusive.
Pettit, R. S., Homer, J., & Gee, R. (2017) · Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education · doi:10.1145/3017680.3017768 · Cited by 66 · cited 6× · see citing works
“lack thereof [50]), this evidence seems to be conflicting [158], but may not be as conflicting as it appears.”
BackgroundMethodResultInfluentialSelf-citationFix the First, Ignore the Rest: Dealing with Multiple Compiler Error Messages
Becker, B. A., Murray, C., Tao, T., Song, C., McCartney, R., & Sanders, K. (2018) · Proceedings of the 49th ACM Technical Symposium on Computer Science Education · doi:10.1145/3159450.3159453 · Cited by 61 · see citing works
Self-citationReplication in computing education research: researcher attitudes and experiences
Ahadi, A., Hellas, A., Ihantola, P., Korhonen, A., & Petersen, A. (2016) · Proceedings of the 16th Koli Calling International Conference on Computing Education Research · doi:10.1145/2999541.2999554 · Cited by 48 · cited 6× · see citing works
“5 let lst = map2 (fun (x, y) -> x + y) [1;2;3] [4;5;6]”
BackgroundInfluentialPUFFT—The Purdue University fast FORTRAN translator
Rosen, S., Spurgeon, R. A., & Donnelly, J. K. (1965) · Communications of the ACM · doi:10.1145/365660.365671 · Cited by 40 · see citing works
Achievement Goals in CS1: Replication and Extension
Zingaro, D., Craig, M., Porter, L., Becker, B. A., Cao, Y., Conrad, P., Cukierman, D., Hellas, A., Loksa, D., & Thota, N. (2018) · Proceedings of the 49th ACM Technical Symposium on Computer Science Education · doi:10.1145/3159450.3159452 · Cited by 35 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401 · see citing works
Self-citationThe Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396 · see citing works
Self-citationPrather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256 · see citing works
Self-citationPrompt Problems: A New Programming Exercise for the Generative AI Era
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630909 · Cited by 187 · see citing works
Self-citationExploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145 · see citing works
Metacognition and Self-Regulation in Programming Education: Theories and Exemplars of Use
Loksa, D., Margulieux, L., Becker, B. A., Craig, M., Denny, P., Pettit, R., & Prather, J. (2022) · ACM Transactions on Computing Education · doi:10.1145/3487050 · Cited by 140 · see citing works
Self-citationOn Designing Programming Error Messages for Novices: Readability and its Constituent Factors
Denny, P., Prather, J., Becker, B. A., Mooney, C., Homer, J., Albrecht, Z. C., & Powell, G. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445696 · Cited by 60 · see citing works
Self-citationError Message Readability and Novice Debugging Performance
Denny, P., Prather, J., & Becker, B. A. (2020) · Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3341525.3387384 · Cited by 50 · see citing works
Self-citationTowards Giving Timely Formative Feedback and Hints to Novice Programmers
Jeuring, J., Keuning, H., Marwan, S., Bouvier, D., Izu, C., Kiesler, N., Lehtinen, T., Lohr, D., Peterson, A., & Sarsa, S. (2022) · Proceedings of the 2022 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3571785.3574124 · Cited by 40 · see citing works
Self-citationSynShine: Improved Fixing of Syntax Errors
Ahmed, T., Ledesma, N. R., & Devanbu, P. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3212635 · Cited by 38 · see citing works
What does saying that 'programming is hard' really say, and about whom?
Becker, B. A. (2021) · Communications of the ACM · doi:10.1145/3469115 · Cited by 33 · see citing works
Self-citationA Comparison of Immediate and Scheduled Feedback in Introductory Programming Projects
Leinonen, J., Denny, P., & Whalley, J. (2022) · Proceedings of the 53rd ACM Technical Symposium on Computer Science Education · doi:10.1145/3478431.3499372 · Cited by 32 · see citing works
Self-citationDecoding Debugging Instruction: A Systematic Literature Review of Debugging Interventions
Yang, S., Baird, M., O’Rourke, E., Brennan, K., & Schneider, B. (2024) · ACM Transactions on Computing Education · doi:10.1145/3690652 · Cited by 29 · see citing works
Promoting Early Engagement with Programming Assignments Using Scheduled Automated Feedback
Denny, P., Whalley, J., & Leinonen, J. (2021) · Proceedings of the 23rd Australasian Computing Education Conference · doi:10.1145/3441636.3442309 · Cited by 24 · see citing works
Self-citationNovice Reflections During the Transition to a New Programming Language
Denny, P., Becker, B. A., Bosch, N., Prather, J., Reeves, B., & Whalley, J. (2022) · Proceedings of the 53rd ACM Technical Symposium on Computer Science Education · doi:10.1145/3478431.3499314 · Cited by 11 · see citing works
Self-citation"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv
Self-citation
- Authors
- Becker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., Pearce, J. L., & Prather, J.
- Affiliations
- University College Dublin (IE)
- University of Auckland (NZ)
- University of Virginia (US)
- Roanoke College (US)
- Southern Illinois University Edwardsville (US)
- University of Toronto Scarborough (CA)
- University of Michigan (US)
- Indian Institute of Technology Kanpur (IN)
- The University of Western Australia (AU)
- Grinnell College (US)
- Berea College (US)
- Abilene Christian University (US)
- ORCID
- Venue
- Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education
- Publisher
- ACM
- Published
- 2019-12-18
- Pages
- 177-210
- OpenAlex
- W3000240292
- Cited by
- 199
- References
- 9
- Citations by year
- 2026: 14 · 2025: 32 · 2024: 43 · 2023: 47 · 2022: 28 · 2021: 21 · 2020: 14
- FWCI
- 18.24
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, programming, and type systems
- Keywords
- compiler error messages, warnings, novice programmers, programming education, student performance prediction, personalized learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
