Do Enhanced Compiler Error Messages Help Students?: Results Inconclusive.
Open accessAbstract
One common frustration students face when first learning to program in a compiled language is the difficulty in interpreting the compiler error messages they receive. Attempts to improve error messages have produced differing results. Two recently published papers showed conflicting results, with one showing measurable change in student behavior, and the other showing no measurable change. We conducted an experiment comparable to these two over the course of several semesters in a CS1 course. This paper presents our results in the context of previous work in this area. We improved the clarity of the compiler error messages the students receive, so that they may more readily understand their mistakes and be able to make effective corrections. Our goal was to help students better understand their syntax mistakes and, as a reasonable measure of our success, we expected to document a decrease in the number of times students made consecutive submissions with the same compilation error. By doing this, we could demonstrate that this enhancement is effective. After collecting and thoroughly analyzing our own experimental data, we found that--despite anecdotal stories, student survey responses, and instructor opinions testifying to the tool's helpfulness--enhancing compiler error messages shows no measurable benefit to students. Our results validate one of the existing studies and contradict another. We discuss some of the reasons for these results and conclude with projections for future research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Instructors spend a considerable amount of time helping novices with cryptic error messages.
“Instructors report that they spend a considerable amount of time helping novices with these often cryptic messages [15,31,32,37].”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Scratch Programming Language and Environment
Maloney, J. H., Resnick, M., Rusk, N., Silverman, B., & Eastmond, E. (2010) · ACM Transactions on Computing Education · doi:10.1145/1868358.1868363 · Cited by 1,423 · see citing works
Storytelling alice motivates middle school girls to learn computer programming
Kelleher, C. L., Pausch, R. F., & Kiesler, S. B. (2007) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/1240624.1240844 · Cited by 505 · see citing works
Identifying and correcting Java programming errors for introductory computer science students
Hristova, M., Misra, A., Rutter, M., & Mercuri, R. T. (2003) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/611892.611956 · Cited by 201 · see citing works
Enhancing syntax error messages appears ineffectual
Denny, P. C., Luxton-Reilly, A., & Carpenter, D. (2014) · doi:10.1145/2591708.2591748 · Cited by 148 · see citing works
A First Look at Novice Compilation Behaviour Using BlueJ
Jadud, M. C. (2005) · Computer Science Education · doi:10.1080/08993400500056530 · Cited by 142 · see citing works
Understanding the syntax barrier for novices
Denny, P., Luxton-Reilly, A., Tempero, E., & Hendrickx, J. (2011) · Proceedings of the 16th annual joint conference on Innovation and technology in computer science education · doi:10.1145/1999747.1999807 · Cited by 133 · see citing works
On Compiler Error Messages: What TheySayand What TheyMean
Traver, V. J. (2010) · Advances in Human-Computer Interaction · doi:10.1155/2010/602570 · Cited by 128 · see citing works
An Effective Approach to Enhancing Compiler Error Messages
Becker, B. A. (2016) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2839509.2844584 · Cited by 125 · see citing works
Identifying and correcting Java programming errors for introductory computer science students
Hristova, M., Misra, A., Rutter, M., & Mercuri, R. T. (2003) · ACM SIGCSE Bulletin · doi:10.1145/792548.611956 · Cited by 120 · see citing works
Murphy, L., Lewandowski, G., McCauley, R. A., Simon, B., Thomas, L., & Zander, C. (2008) · ACM SIGCSE Bulletin · doi:10.1145/1352322.1352191 · Cited by 118 · see citing works
Measuring the effectiveness of error messages designed for novice programmers
Marceau, G., Fisler, K., & Krishnamurthi, S. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953308 · Cited by 98 · see citing works
Murphy, L., Lewandowski, G., McCauley, R. A., Simon, B., Thomas, L., & Zander, C. (2008) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1352135.1352191 · Cited by 89 · see citing works
BlueFix: Using Crowd-Sourced Feedback to Support Programming Students in Error Diagnosis and Repair
Watson, C., Li, F. W. B., & Godwin, J. L. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33642-3_25 · Cited by 59 · see citing works
Schorsch, T. (1995) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/199688.199769 · Cited by 59 · see citing works
Empowering students and building confidence in novice programmers through gauntlet
Flowers, T. R., Carver, C. A., & Jackson, J. (2005) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2004.1408551 · Cited by 56 · see citing works
Freund, S. N., & Roberts, E. S. (1996) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/236452.236560 · Cited by 48 · see citing works
A detector for non-literal Java errors
Dy, T. T., & Rodrigo, M. M. T. (2010) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930485 · Cited by 28 · see citing works
Schorsch, T. (1995) · ACM SIGCSE Bulletin · doi:10.1145/199691.199769 · Cited by 24 · see citing works
Barik, T., Witschey, J., Johnson, B., & Murphy-Hill, E. R. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2591062.2591124 · Cited by 21 · see citing works
Freund, S. N., & Roberts, E. S. (1996) · ACM SIGCSE Bulletin · doi:10.1145/236462.236560 · Cited by 14 · see citing works
Carter, E. J. (2015) · Journal of computing sciences in colleges · Cited by 11 · see citing works
LearnCS!: a new, browser-based C programming environment for CS1
Lipman, D. (2014) · Journal of computing sciences in colleges · Cited by 8 · see citing works
Hasker, R. W. (2002) · Journal of computing sciences in colleges · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Christopher Watson, Frederick W. B. Li, and Jamie L. Godwin. 2012. BlueFix: using crowd-sourced feedback to support programming students in error diagnosis and repair. In Proceedings of the 11th international conference on Advances in Web-Based Learning (ICWL'12), Elvira
Cited by
Sorted by citations
Introductory programming: a systematic literature review
Luxton-Reilly, A., Simon, Al-Bluwi, I., Becker, B. A., Giannakos, M. N., Kumar, A. N., Ott, L. M., Paterson, J. H., Scott, M., Sheard, J., et al. (2018) · doi:10.1145/3293881.3295779 · Cited by 511 · see citing works
Using Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235 · see citing works
Becker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372508 · Cited by 199 · see citing works
“lack thereof [50]), this evidence seems to be conflicting [158], but may not be as conflicting as it appears.”
BackgroundMethodResultInfluentialSelf-citationMetacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools
Prather, J., Pettit, R., McMurry, K., Peters, A., Homer, J., & Cohen, M. (2018) · Proceedings of the 2018 ACM Conference on International Computing Education Research · doi:10.1145/3230977.3230981 · Cited by 129 · see citing works
“Pettit et al. enhanced CEMs in an automated assessment tool, Athene, used for C++ programming and did not find conclusive results that the ECEMs were more helpful than standard CEMs [50].”
MethodSelf-citation(2019) · doi:10.1145/3287324.3287374 · Cited by 104 · see citing works
“A few researchers have reported empirical results on the efficacy of ECEMs, though the efficacy of this approach is an ongoing debate [6, 7, 11, 35, 37].”
BackgroundSelf-citationOn Novices' Interaction with Compiler Error Messages: A Human Factors Approach
Prather, J., Pettit, R., McMurry, K. H., Peters, A., Homer, J., Simone, N., & Cohen, M. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106169 · Cited by 72 · see citing works
“One research tool often employed in evaluating changes made to CS1 classes is the think-aloud protocol where students are observed writing code and asked to verbalize their thoughts while doing so.”
BackgroundMethodInfluentialSelf-citationCommon logic errors made by novice programmers
Ettles, A., Luxton-Reilly, A., & Denny, P. C. (2018) · Australasian Computing Education Conference · doi:10.1145/3160489.3160493 · Cited by 70 · see citing works
“In particular, recent work by Becker [3–5], Denny [7, 8], Pettit [13], and Prather [14] examined common syntax errors and evaluated strategies, such as improving the messages generated by the compiler, for helping students to resolve them.”
BackgroundProgramming Paradigms and Beyond
Krishnamurthi, S., & Fisler, K. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.014 · Cited by 65 · see citing works
The Effects of Enhanced Compiler Error Messages on a Syntax Error Debugging Test
Becker, B. A., Goslin, K., & Glanville, G. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159461 · Cited by 62 · see citing works
On Designing Programming Error Messages for Novices: Readability and its Constituent Factors
Denny, P., Prather, J., Becker, B. A., Mooney, C., Homer, J., Albrecht, Z. C., & Powell, G. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445696 · Cited by 60 · see citing works
“Eforts focusing on enhancing/improving compiler error messages have been made by researchers such as Barik [4], Becker [13], Denny [19], Kohn [33], Pettit [45], Prather [46], and Karkare [2].”
BackgroundSelf-citationA Closer Look at Metacognitive Scaffolding
Denny, P. C., Prather, J. E., Becker, B. A., Albrecht, Z., Loksa, D., & Pettit, R. (2019) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3364510.3366170 · Cited by 49 · see citing works
Self-citationStatic Analyses in Python Programming Courses
Liu, D., & Petersen, A. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287503 · Cited by 46 · see citing works
“performed similar work at their institution and concur, stating that despite positive feedback from both students and instructors, they saw no measurable benefit in terms of a reduction in student submissions to programming problems [13].”
BackgroundQian, Y., & Lehman, J. D. (2019) · Sage Open · doi:10.1177/2158244019885136 · Cited by 42 · see citing works
“The other type of feedback system uses manually designed feedback messages for common student errors identified using the student data in the automated assessment system (Becker, 2016; Denny, Luxton-Reilly, & Carpenter, 2014; Pettit et al., 2017).”
BackgroundMethodResultInfluentialTargeted Example Generation for Compilation Errors
Ahmed, U. Z., Sindhgatta, R., Srivastava, N., & Karkare, A. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00039 · Cited by 36 · see citing works
“In contrast, other studies have found that additional information does not help students significantly [30], [32], and that placement and structuring of the message is more crucial [14].”
ResultA Large Scale RCT on Effective Error Messages in CS1
Wang, S., Mitchell, J. C., & Piech, C. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630764 · Cited by 35 · see citing works
“Despite numerous efforts on improving error messages, past studies yield varied results on the effectiveness of different approaches [8, 10, 14, 16, 32, 37].”
ResultKazemitabaar, M., Huang, O., Suh, S., Henley, A. Z., & Grossman, T. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712104 · Cited by 48 · see citing works
“This section discusses the challenges faced by novice programmers, established interventions that address these difficulties, and the evolving role of Large Language Models (LLMs) in programming education, including their potential impact on metacognitive skills and over-reliance on AI.”
Taylor, A., Vassar, A., Renzella, J., & Pearce, H. A. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630822 · Cited by 33 · see citing works
“Initial work has been done to enhance the readability of compiler error messages and explore the use of enhanced error messages [6–8, 10, 13, 27].”
BackgroundAhmed, U. Z., Srivastava, N., Sindhgatta, R., & Karkare, A. (2020) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3377814.3381703 · Cited by 33 · see citing works
“Studies from multiple labs have documented null results from the use of such static text message enhancements [12, 29, 31], while some have claimed positive outcomes [4, 6].”
BackgroundMethodInfluentialEnhancing Programming Error Messages in Real Time with Generative AI
Kimmel, B., Geisert, A. L., Yaro, L., Gipson, B., Hotchkiss, R., Osae-Asante, S. K., Vaught, H., Wininger, G., & Yamaguchi, C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3647967 · Cited by 29 · see citing works
“Many studies have been done on enhancing the error messages served to students through Athene [18, 20–22].”
Taipalus, T., Grahn, H., & Ghanbari, H. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111034 · Cited by 24 · see citing works
“…spacing, colors (Dong and Khandwala, 2019), and syntax highlighting (Hannebauer et al., 2018) have been shown to increase error message effectiveness, the evidence has sometimes been anecdotal (Becker et al., 2019), or even controversial (Sarkar, 2015; Denny et al., 2014; Pettit et al., 2017).”
BackgroundLearning From Errors: Exploring the Effectiveness of Enhanced Error Messages in Learning to Program
Zhou, Z., Wang, S.-J., & Qian, Y. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.768962 · Cited by 24 · see citing works
“, 2012), compiler errors (Pettit et al., 2017), novice programmer errors (McCall and Kölling, 2019), and others (see Becker et al.”
BackgroundInfluentialBuilding a Corpus of Task-Based Grading and Feedback Systems for Learning and Teaching Programming
Strickroth, S., & Striewe, M. (2022) · International Journal of Engineering Pedagogy (iJEP) · doi:10.3991/ijep.v12i5.31283 · Cited by 20 · see citing works
Kohn, T., & Manaris, B. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366920 · Cited by 18 · see citing works
An Inquisitive Code Editor for Addressing Novice Programmers' Misconceptions of Program Behavior
Henley, A. Z., Ball, J., Klein, B., Rutter, A., & Lee, D. (2021) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1109/icse-seet52601.2021.00026 · Cited by 18 · see citing works
An Investigation of High School Students’ Errors in Introductory Programming: A Data-Driven Approach
Qian, Y., & Lehman, J. D. (2019) · Journal of Educational Computing Research · doi:10.1177/0735633119887508 · Cited by 18 · see citing works
“Most previous studies on student errors in introductory programming examined Java compilation errors, but a few studies used AATs to explore common student errors in other programming languages such as Python (Qian, Yan, & Zhou, 2019), C þþ (Pettit et al., 2017), and Pascal (Qian & Lehman, 2018).”
BackgroundResultJemmali, C., Kleinman, E., Bunian, S., Almeda, M. V., Rowe, E. A., & El‐Nasr, M. S. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366824 · Cited by 18 · see citing works
Software Analytics to Support Students in Object-Oriented Programming Tasks: An Empirical Study
Ardimento, P., Bernardi, M. L., & Cimitile, M. (2020) · IEEE Access · doi:10.1109/access.2020.3010172 · Cited by 17 · see citing works
“Focusing on syntax [12], [13] some authors enhance compiler error messages [14]–[17], logic errors [18], [19], and coding habits [20]–[23].”
BackgroundNip, T., Gunter, E. L., Herman, G. L., Morphew, J., & West, M. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159500 · Cited by 17 · see citing works
“…education have focused on designing new pedagogies (e.g., pair programming [32] and CS unplugged [30]) or new pedagogical tools (e.g., block languages [36] and informative compiler error messages [28]), comparatively little research has focused on re-designing traditional models of assessment.”
BackgroundEastwood-Tidy: C Linting for Automated Code Style Assessment in Programming Courses
Hart, R., Hays, B., McMillin, C., Rezig, E. K., Rodriguez-Rivera, G., & Turkstra, J. A. (2023) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545945.3569817 · Cited by 16 · see citing works
“Research has been conducted into the effectiveness of improved, human-readable compiler errors with mixed [24] to positive results [3] measured in increased student performance by Pettit et al.”
BackgroundNot the Silver Bullet: LLM-enhanced Programming Error Messages are Ineffective in Practice
Santos, E. A., & Becker, B. A. (2024) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3689535.3689554 · Cited by 15 · see citing works
“There have been many attempts at having programming systems produce better diagnostics [9, 20, 25, 37, 40], however, error message enhancement has seen weak [3, 6, 15, 34] to insignificant [11, 32] results in improving student outcomes.”
Analysis of a Process for Introductory Debugging
Whalley, J. L., Settle, A., & Luxton-Reilly, A. (2021) · Australasian Computing Education Conference · doi:10.1145/3441636.3442300 · Cited by 13 · see citing works
“More recently, this work has focused on analysing large repositories of naturally occurring data generated through compilation events [1, 5, 8, 13–15, 28, 44, 53, 55, 56].”
BackgroundQian, Y., & Lehman, J. D. (2021) · Journal of Research on Technology in Education · doi:10.1080/15391523.2020.1865220 · Cited by 12 · see citing works
Edwards, S. H., Spacco, J. W., & Hovemeyer, D. (2019) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2019.941 · Cited by 12 · see citing works
Framework for SQL Error Message Design: A Data-Driven Approach
Taipalus, T., & Grahn, H. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3607180 · Cited by 11 · see citing works
“For example, some studies have tested enhanced error messages with novices, yet concluded that there are no positive efects [15, 40].”
BackgroundBecker, B. A., Denny, P. C., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3304221.3325539 · Cited by 11 · see citing works
“This is exacerbated by the fact that although some works have presented evidence on diagnostic message effectiveness [1] (or lack thereof [7]), this evidence seems to be conflicting [11].”
BackgroundSelf-citationCategorization of Frequent Errors in Solution Codes Created by Novice Programmers
Rahman, M. M., Kawabayashi, S., & Watanobe, Y. (2021) · SHS Web of Conferences · doi:10.1051/shsconf/202110204014 · Cited by 11 · see citing works
Rigby, L., Denny, P. C., & Luxton-Reilly, A. (2020) · Australasian Computing Education Conference · doi:10.1145/3373165.3373169 · Cited by 10 · see citing works
Towards Assessing the Readability of Programming Error Messages
Becker, B. A., Denny, P. C., Prather, J. E., Pettit, R., Nix, R., & Mooney, C. (2021) · Australasian Computing Education Conference · doi:10.1145/3441636.3442320 · Cited by 10 · see citing works
“‘Improved’ or ‘enhanced’ error messages have been incorporated into many tools such as novice code editors and automated assessment tools [3, 4, 8, 16, 19, 28, 32, 33, 41], often with positive effects such as reducing the number of errors overall [5].”
BackgroundSelf-citationPaiva, J. C., Figueira, Á., & Leal, J. P. (2023) · Electronics · doi:10.3390/electronics12102254 · Cited by 9 · see citing works
Using Data to Understand Difficulties of Learning to Program
Qian, Y., Yan, P., & Zhou, M.-K. (2019) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3300115.3309521 · Cited by 9 · see citing works
“An automated assessment system is a tool that can automatically evaluate the correctness of students’ programs [9, 17].”
MethodFirst Steps Towards Predicting the Readability of Programming Error Messages
Prather, J. E., Denny, P. C., Becker, B. A., Nix, R., Reeves, B. N., Randrianasolo, A. S., & Powell, G. (2023) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545945.3569791 · Cited by 8 · see citing works
“An unfortunate side effect of poorly written messages is that educators spend an inordinate amount of time helping novices deal with PEMs [4, 28, 36].”
BackgroundA Mixed-Methods Study of Novice Programmer Interaction with Python Error Messages
D'souza, R., Bhayana, M., Ahmadzadeh, M., & Harrington, B. (2019) · Western Canadian Conference on Computing Education · doi:10.1145/3314994.3325090 · Cited by 6 · see citing works
“While Petit et al.[10], were unable to find a significant benefit of EEMs, they did receive positive feedback from both students and instructors.”
BackgroundIntegrating programming errors into knowledge graphs for automated assignment of programming tasks
Ding, G., Shi, X., & Li, S. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-12026-7 · Cited by 6 · see citing works
“Pettit et al. (2017) and Becker et al. (2018) proposed methods to improve the accuracy of compiler error messages since beginners often failed to understand the error messages due to the complex user interfaces of programming tools.”
Helping Student Programmers Through Industrial-Strength Static Analysis
Senger, A., Edwards, S. H., & Ellis, M. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3478431.3499310 · Cited by 5 · see citing works
“Denny et al. [3], Pettit et al. [10], and Becker [1] examine how compiler messages could be improved in a teaching context because compiler errors are usually designed for professional developers.”
BackgroundSánchez‐Santillán, M., Fernandez‐Medina, C., Pérez, J. R. P., & Ruiz, M. P. P. (2023) · Computer Applications in Engineering Education · doi:10.1002/cae.22640 · Cited by 4 · see citing works
Does Static Analysis Help Software Engineering Students?
Plösch, R., & Neumüller, C. (2020) · International Conference on Educational and Information Technology · doi:10.1145/3383923.3383957 · Cited by 4 · see citing works
Building Blocks Towards More Effective SQL Error Messages
Taipalus, T., & Grahn, H. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653552 · Cited by 3 · see citing works
Error Messages for Students Taught Using a Systematic Program Design Curriculum
Dzhatdoyev, S., Rosiers, J. A. D., & Morazán, M. T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-74558-4_9 · Cited by 3 · see citing works
Automatic Feedback in the Teaching of Programming in Undergraduate Courses: a Literature Mapping
Conceicao, W., de Holanda, M. T., Macedo, F., Ishikawa, E., Nunes, V., & Silva, D. D. (2022) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie56618.2022.9962723 · Cited by 3 · see citing works
A replication study on the intuitiveness of programming language syntax
Lappi, V., Tirronen, V., & Itkonen, J. (2023) · Software Quality Journal · doi:10.1007/s11219-023-09631-7 · Cited by 6 · see citing works
“This finding was replicated by Pettit et al. (2017), though a study by Becker (2016) had the opposite results.”
ResultSolano, L. L., Renzella, J., & Vassar, A. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626253.3635483 · Cited by 3 · see citing works
“As such, the approaches to error message enhancement have varied significantly in method, and have often found inconclusive results [14].”
BackgroundNovice Perceptions on Effective Elements of PostgreSQL Error Messages
Taipalus, T., Grahn, H., Ritonummi, S., Siitonen, V., Vartiainen, T., & Zhidkikh, D. (2025) · ACM Transactions on Computing Education · doi:10.1145/3732790 · Cited by 3 · see citing works
Prather, J. E., Pettit, R., Becker, B. A., Denny, P. C., Loksa, D., Peters, A., Albrecht, Z., & Masci, K. (2019) · ACM Inroads · doi:10.1145/3324892 · Cited by 2 · see citing works
“They conducted a think-aloud study with novice programmers solving a programming problem while using Athene, an AAT with a long history of empirical work [33, 35–37 , 45 ].”
MethodSelf-citationD'souza, R., Bernuy, A. Z., & Harrington, B. (2021) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3408877.3439652 · Cited by 2 · see citing works
“Several studies have attempted to assess the effect of varying EMs on students with mixed results [2, 3, 5].”
Qian, Y., & Wu, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3454643 · Cited by 1 · see citing works
“This may also explain the ineffectiveness of using data-driven feedback in relevant studies [6], [9], [10].”
BackgroundMethodResultInfluentialEnhanced SQL error messages facilitate faster error fixing
Taipalus, T., Grahn, H., & Knutas, A. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10695-1 · Cited by 1 · see citing works
“However, several studies have also presented inconclusive or negative outcomes (e.g., Prather et al. 2017; Nienaltowski et al. 2008; Pettit et al. 2017).”
Mind the Error Message: An Inverted Quiz Format to Direct Learner's Attention to Error Messages
Tsunoda, K., Masuhara, H., & Cong, Y. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588823 · Cited by 1 · see citing works
“However, some research [8, 25, 26, 31] could not confirm a positive effect of enhancing error messages like the above.”
BackgroundMethodOliveira, J. D., Oliveira, E., de Carvalho, L. S. G., & Oliveira, D. B. F. (2019) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/cbie.sbie.2019.329 · Cited by 1 · see citing works
Investigating Compilation Errors of Students Learning Haskell
Németh, B., Choi, E., Makihara, E., & Iida, H. (2019) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.295.4
“Pettit and Gee report in their paper that by simply making more useful and customized error messages for certain common programmer mistakes will not make students less likely to repeat the same errors [6], but their results are not conclusive.”
BackgroundToward Productivity Improvements in Programming Languages Through Behavioral Analytics
Daleiden, P. (2020) · Digital Scholarship - UNLV (University of Nevada Reno) · doi:10.34917/19412056
Programmers' Views on IDE Compilation Mechanisms
Karvelas, I., Dillane, J., & Becker, B. A. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617915
Unlocking Student Potential With TA-Bot: Timely Submissions and Improved Code Style
Forden, J., Schneider, M. P., Gebhard, A., Molla, M. T. I., & Brylow, D. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701955
A Design Recipe and Recipe-Based Errors for Regular Expressions
Morazán, M. T., Dzhatdoyev, S., Rosiers, J. A. D., Minić, T., Garced, A. M., & Fields, D. A. K. (2025) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.424.2
“One study found that longer error messages are not more effective [47] while another study was inconclusive [48].”
Santos, E. A., Salmon, A., & Hammer, K. (2025) · IEEE Access · doi:10.1109/access.2025.3613500
“Evaluations of traditional PEM enhancement has yielded weak [13], [15] or mixed [9], [16] results.”
Exploring the quality and effectiveness of AI-generated feedback in introductory programming
Qian, Y., Liu, M., & Zhu, L. (2026) · Humanities and Social Sciences Communications · doi:10.1057/s41599-026-07920-7
“While error message enhancement and data-driven feedback both provide elaborated feedback messages for syntax errors, most relevant studies reported ineffectiveness for improving students ’ learning performance in programming courses (Denny et al. 2014; Pettit et al. 2017; Zhou et al. 2021).”
Dong, X., & Lin, G. Y. (2025) · International Symposium on Artificial Intelligence for Education (ISAIE) · doi:10.1145/3775073.3775185
“Another widely reported automated approach is the enhancement of error messages [ 4 ][ 5 ][ 6 ] , which has attracted sustained attention for several decades, yet its effectiveness remains inconclusive [ 7 ][ 8 ] .”
- Authors
- Pettit, R. S., Homer, J., & Gee, R.
- Affiliations
- Abilene Christian University (US)
- ORCID
- Pettit, Raymond S.
- Venue
- Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education
- Publisher
- ACM
- Published
- 2017-03-08
- Pages
- 465-470
- OpenAlex
- W2604953184
- Cited by
- 66
- References
- 23
- Citations by year
- 2026: 1 · 2025: 8 · 2024: 7 · 2023: 10 · 2022: 3 · 2021: 9 · 2020: 7 · 2019: 15 · 2018: 5 · 2017: 1
- FWCI
- 7.06
- Type
- conference paper
- Language
- en
- Open access
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion)
- Digital Commons - ACU (Abilene Christian University) (submittedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, programming, and type systems
- Keywords
- compiler error messages, CS1, introductory programming, syntactic errors, programming education, compilation errors, student behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
