On Novices' Interaction with Compiler Error Messages: A Human Factors Approach
Open accessAbstract
The difficulty in understanding compiler error messages can be a major impediment to novice student learning. To alleviate this issue, multiple researchers have run experiments enhancing compiler error messages in automated assessment tools for programming assignments. The conclusions reached by these published experiments appear to be conducting. We examine these experiments and propose five potential reasons for the inconsistent conclusions concerning enhanced compiler error messages: (1) students do not read them, (2) researchers are measuring the wrong thing, (3) the effects are hard to measure, (4) the messages are not properly designed, (5) the messages are properly designed, but students do not understand them in context due to increased cognitive load. We constructed mixed-methods experiments designed to address reasons 1 and 5 with a specific automated assessment tool, Athene, that previously reported inconclusive results. Testing student comprehension of the enhanced compiler error messages outside the context of an automated assessment tool demonstrated their effectiveness over standard compiler error messages. Quantitative results from a 60 minute one-on-one think-aloud study with 31 students did not show substantial increase in student learning outcomes over the control. However, qualitative results from the one-on-one think-aloud study indicated that most students are reading the enhanced compiler error messages and generally make effective changes after encountering them.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous work shows that a system providing enhanced feedback specifically targeted at novice programmers can increase efficacy and metacognitive behaviors.
“Previous work shows that a system providing enhanced feedback specifically targeted at novice programmers can increase efficacy [29, 72] and metacognitive behaviors [70].”
From “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers (2024) · cites this paper as [72]
Citation [70] is attached to the claim about increased metacognitive behaviors. The Conclusions report that novice students read enhanced compiler error messages and understood them better, but do not report an increase in metacognitive behaviors. Section 2.4 discusses metacognitive-awareness training in a separate study and calls it beyond this paper’s scope. (LLM, conf=0.73)
Experimental — accuracy not yet measuredProgramming error messages can be notoriously difficult to decipher, especially for novices.
“Programming Error Messages (PEMs) can be notoriously difficult to decipher, especially for novices [33]”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
(2014) · Medical Entomology and Zoology · Cited by 6,828 · see citing works
Research Design: Qualitative, Quantitative, and Mixed Methods Approaches
Perreault, K. (2010) · Manual Therapy · doi:10.1016/j.math.2010.09.003 · Cited by 4,875 · see citing works
Twycross, A. M. (2004) · Nurse Researcher · doi:10.7748/nr.12.1.82.s2 · Cited by 2,032 · see citing works
Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests
Rubin, J. N. (1994) · Cited by 1,750 · see citing works
The Scratch Programming Language and Environment
Maloney, J. H., Resnick, M., Rusk, N., Silverman, B., & Eastmond, E. (2010) · ACM Transactions on Computing Education · doi:10.1145/1868358.1868363 · Cited by 1,423 · see citing works
Failure rates in introductory programming
Bennedsen, J., & Caspersen, M. E. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1272848.1272879 · Cited by 634 · see citing works
Review of recent systems for automatic assessment of programming assignments
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930480 · Cited by 561 · see citing works
A Survey of Automated Assessment Approaches for Programming Assignments
Ala-Mutka, K. (2005) · Computer Science Education · doi:10.1080/08993400500150747 · Cited by 550 · see citing works
Storytelling alice motivates middle school girls to learn computer programming
Kelleher, C. L., Pausch, R. F., & Kiesler, S. B. (2007) · ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) · doi:10.1145/1240624.1240844 · Cited by 505 · see citing works
Automatic test-based assessment of programming
Douce, C., Livingstone, D., & Orwell, J. (2005) · Journal on Educational Resources in Computing · doi:10.1145/1163405.1163409 · Cited by 416 · see citing works
What would other programmers do
Hartmann, B., MacDougall, D., Brandt, J., & Klemmer, S. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753478 · Cited by 280 · see citing works
Edwards, S. H., & Pérez-Quiñones, M. A. (2008) · Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1384271.1384371 · Cited by 256 · see citing works
Automatic graders for programming classes
Hollingsworth, J. (1960) · Communications of the ACM · doi:10.1145/367415.367422 · Cited by 231 · see citing works
What developers want and need from program analysis: an empirical study
Christakis, M., & Bird, C. (2016) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2970276.2970347 · Cited by 231 · see citing works
Programming, Problem Solving, and Self-Awareness: Effects of Explicit Guidance
Loksa, D., Ko, A. J., Jernigan, W., Oleson, A., Mendez, C. J., & Burnett, M. M. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858252 · Cited by 223 · see citing works
Identifying and correcting Java programming errors for introductory computer science students
Hristova, M., Misra, A., Rutter, M., & Mercuri, R. T. (2003) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/611892.611956 · Cited by 201 · see citing works
Enhancing syntax error messages appears ineffectual
Denny, P. C., Luxton-Reilly, A., & Carpenter, D. (2014) · doi:10.1145/2591708.2591748 · Cited by 148 · see citing works
On Compiler Error Messages: What TheySayand What TheyMean
Traver, V. J. (2010) · Advances in Human-Computer Interaction · doi:10.1155/2010/602570 · Cited by 128 · see citing works
Personifying programming tool feedback improves novice programmers' learning
Lee, M. J., & Ko, A. J. (2011) · Proceedings of the seventh international workshop on Computing education research · doi:10.1145/2016911.2016934 · Cited by 125 · see citing works
An Effective Approach to Enhancing Compiler Error Messages
Becker, B. A. (2016) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2839509.2844584 · Cited by 125 · see citing works
Identifying Top Java Errors for Novice Programmers
Jackson, J., Cobb, M. L., & Carver, C. A. (2006) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2005.1611967 · Cited by 100 · see citing works
Measuring the effectiveness of error messages designed for novice programmers
Marceau, G., Fisler, K., & Krishnamurthi, S. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953308 · Cited by 98 · see citing works
Murphy, L., Lewandowski, G., McCauley, R. A., Simon, B., Thomas, L., & Zander, C. (2008) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1352135.1352191 · Cited by 89 · see citing works
Do Developers Read Compiler Error Messages?
Barik, T., Smith, J., Lubick, K., Holmes, E., Feng, J., Murphy-Hill, E., & Parnin, C. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.59 · Cited by 86 · cited 3× · see citing works
“[2] use eye tracking software to examine whether intermediate students read error messages.”
MethodNienaltowski, M.-H., Pedroni, M., & Meyer, B. (2008) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1352135.1352192 · Cited by 78 · see citing works
Marceau, G., Fisler, K., & Krishnamurthi, S. (2011) · doi:10.1145/2048237.2048241 · Cited by 77 · see citing works
Do Enhanced Compiler Error Messages Help Students?: Results Inconclusive.
Pettit, R. S., Homer, J., & Gee, R. (2017) · Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education · doi:10.1145/3017680.3017768 · Cited by 66 · cited 14× · see citing works
“One research tool often employed in evaluating changes made to CS1 classes is the think-aloud protocol where students are observed writing code and asked to verbalize their thoughts while doing so.”
BackgroundMethodInfluentialSelf-citationA qualitative think-aloud study of novice programmers' code writing strategies
Whalley, J. L., & Kasto, N. (2014) · Annual Conference on Innovation and Technology in Computer Science Education · doi:10.1145/2591708.2591762 · Cited by 61 · see citing works
BlueFix: Using Crowd-Sourced Feedback to Support Programming Students in Error Diagnosis and Repair
Watson, C., Li, F. W. B., & Godwin, J. L. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33642-3_25 · Cited by 59 · see citing works
Schorsch, T. (1995) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/199688.199769 · Cited by 59 · see citing works
Empowering students and building confidence in novice programmers through gauntlet
Flowers, T. R., Carver, C. A., & Jackson, J. (2005) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2004.1408551 · Cited by 56 · see citing works
An Empirical Study of Iterative Improvement in Programming Assignments
Pettit, R., Homer, J., Gee, R., et al. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677279 · Cited by 54 · see citing works
Self-citationFreund, S. N., & Roberts, E. S. (1996) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/236452.236560 · Cited by 48 · see citing works
Teaching Java with the BlueJ Environment
Hagan, D., & Markham, S. (2000) · Cited by 32 · see citing works
A qualitative think aloud study of the early neo-piagetian stages of reasoning in novice programmers
Teague, D., Corney, M., Ahadi, A., et al. (2013) · QUT ePrints (Queensland University of Technology) · doi:10.5555/2667199.2667209 · Cited by 31 · see citing works
A detector for non-literal Java errors
Dy, T. T., & Rodrigo, M. M. T. (2010) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930485 · Cited by 28 · see citing works
Barik, T., Witschey, J., Johnson, B., & Murphy-Hill, E. R. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2591062.2591124 · Cited by 21 · see citing works
Pettit, R., & Prather, J. E. (2017) · Journal of computing sciences in colleges · doi:10.5555/3055338.3079060 · Cited by 19 · see citing works
Self-citationWexelblat, R. L. (1976) · doi:10.5555/800253.807695 · Cited by 17 · see citing works
Freund, S. N., & Roberts, E. S. (1996) · ACM SIGCSE Bulletin · doi:10.1145/236462.236560 · Cited by 14 · see citing works
Novices' knowledge construction of difficult concepts in CS1
Yuen, T. T. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1345375.1345413 · Cited by 11 · see citing works
Carter, E. J. (2015) · Journal of computing sciences in colleges · Cited by 11 · see citing works
Towell, D., & Reeves, B. N. (2010) · Digital Commons - ACU (Abilene Christian University) · Cited by 9 · see citing works
LearnCS!: a new, browser-based C programming environment for CS1
Lipman, D. (2014) · Journal of computing sciences in colleges · Cited by 8 · see citing works
Hasker, R. W. (2002) · Journal of computing sciences in colleges · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 7
- Stephen H Edwards and Manuel A Perez-Quinones. 2008. Web-CAT: automati- cally grading programming assignments. In ACM SIGCSE Bulletin, Vol. 40. ACM, 328-328.
- Karl Anders Ericsson and Herbert Alexander Simon. 1993. Protocol analysis. MIT press Cambridge, MA.
- Stephen N Freund and Eric S Roberts. 1996. Thetis: an ANSI C programming environment designed for introductory use. In SIGCSE, Vol. 96. 300-304.
- Yvonna S Lincoln and Egon G Guba. 1985. Naturalistic inquiry. Vol. 75. Sage.
- Laurie Murphy, Gary Lewandowski, Renée McCauley, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: the good, the bad, and the quirky-a qualitative analysis of novices' strategies. In ACM SIGCSE Bulletin, Vol. 40. ACM, 163-167.
- Marie-Hélène Nienaltowski, Michela Pedroni, and Bertrand Meyer. 2008. Com- piler error messages: What can help novices?. In ACM SIGCSE Bulletin, Vol. 40. ACM, 168-172.
- Tom Schorsch. 1995. CAP: an automated self-assessment tool to check Pascal programs for syntax, logic and style errors. In ACM SIGCSE Bulletin, Vol. 27. ACM, 168-172.
Cited by
Sorted by citations
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256 · see citing works
“Previous work shows that a system providing enhanced feedback specifically targeted at novice programmers can increase efficacy [29, 72] and metacognitive behaviors [70].”
BackgroundSelf-citationThe Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196 · see citing works
“Next, the participant was given a warm-up task because following a think-aloud protocol can be difficult to do while also doing cognitively demanding tasks, as suggested by [53, 63, 67].”
Self-citationMetacognition and Self-Regulation in Programming Education: Theories and Exemplars of Use
Loksa, D., Margulieux, L., Becker, B. A., Craig, M., Denny, P., Pettit, R., & Prather, J. (2022) · ACM Transactions on Computing Education · doi:10.1145/3487050 · Cited by 140 · see citing works
Self-citationMetacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools
Prather, J., Pettit, R., McMurry, K., Peters, A., Homer, J., & Cohen, M. (2018) · Proceedings of the 2018 ACM Conference on International Computing Education Research · doi:10.1145/3230977.3230981 · Cited by 129 · see citing works
Self-citationCognitive Load Theory in Computing Education Research: A Review
Duran, R., Zavgorodniaia, A., & Sorva, J. (2022) · ACM Transactions on Computing Education · doi:10.1145/3483843 · Cited by 92 · see citing works
Fix the First, Ignore the Rest: Dealing with Multiple Compiler Error Messages
Becker, B. A., Murray, C., Tao, T., Song, C., McCartney, R., & Sanders, K. (2018) · Proceedings of the 49th ACM Technical Symposium on Computer Science Education · doi:10.1145/3159450.3159453 · Cited by 61 · see citing works
On Designing Programming Error Messages for Novices: Readability and its Constituent Factors
Denny, P., Prather, J., Becker, B. A., Mooney, C., Homer, J., Albrecht, Z. C., & Powell, G. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445696 · Cited by 60 · see citing works
Self-citationThe Error Behind The Message: Finding the Cause of Error Messages in Python
Kohn, T. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287381 · Cited by 41 · see citing works
Rubric Is All You Need: Enhancing LLM-based Code Evaluation With Question-Specific Rubrics
Pathak, A., Gandhi, R., Uttam, V., Ramamoorthy, A., Ghosh, P., Jindal, A. R., Verma, S., Mittal, A., Ased, A., Khatri, C., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.23989 · Cited by 30
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B., Kimmel, B., Wright, J., & Briggs, B. (2024) · arXiv · Cited by 196 · see citing works
“Next, the participant was given a warm-up task because following a think-aloud protocol can be difficult to do while also doing cognitively demanding tasks, as suggested by [53, 63, 67].”
Self-citation"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv
“Previous work shows that a system providing enhanced feedback specifically targeted at novice programmers can increase efficacy [29, 72] and metacognitive behaviors [70].”
BackgroundSelf-citation
- Authors
- Prather, J., Pettit, R., McMurry, K. H., Peters, A., Homer, J., Simone, N., & Cohen, M.
- Affiliations
- Abilene Christian University (US)
- Nova Southeastern University (US)
- Venue
- Proceedings of the 2017 ACM Conference on International Computing Education Research
- Publisher
- ACM
- Published
- 2017-08-14
- Pages
- 74-82
- OpenAlex
- W2747427841
- Cited by
- 72
- References
- 45
- Citations by year
- 2026: 1 · 2025: 11 · 2024: 11 · 2023: 14 · 2022: 7 · 2021: 7 · 2020: 5 · 2019: 10 · 2018: 6
- FWCI
- 5.29
- Type
- conference paper
- Language
- en
- Open access
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion)
- Digital Commons - ACU (Abilene Christian University) (submittedVersion, cc-by)
- Funders
- Abilene Christian University
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- compiler error messages, novice programmers, automated assessment systems, programming education, Athena, cognitive load, think-aloud study, learning outcomes, programming exercises
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
