On Designing Programming Error Messages for Novices: Readability and its Constituent Factors
Open accessAbstract
Programming error messages play an important role in learning to program. The cycle of program input and error message response completes a loop between the programmer and the compiler/interpreter and is a fundamental interaction between human and computer. However, error messages are notoriously problematic, especially for novices. Despite numerous guidelines citing the importance of message readability, there is little empirical research dedicated to understanding and assessing it. We report three related experiments investigating factors that influence programming error message readability. In the first two experiments we identify possible factors, and in the third we ask novice programmers to rate messages using scales derived from these factors. We find evidence that several key factors significantly affect message readability: message length, jargon use, sentence structure, and vocabulary. This provides novel empirical support for previously untested long-standing guidelines on message design, and informs future efforts to create readability metrics for programming error messages.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous work shows that a system providing enhanced feedback specifically targeted at novice programmers can increase efficacy and metacognitive behaviors.
“Previous work shows that a system providing enhanced feedback specifically targeted at novice programmers can increase efficacy [29, 72] and metacognitive behaviors [70].”
From “It’s Weird That it Knows What I Want”: Usability and Interactions with Copilot for Novice Programmers (2024) · cites this paper as [29]
Section 2.4 reports that changes to programming error messages increased usability, measured by comprehension and error-resolution rates, but cautions that evidence for the effectiveness of enhanced compiler error messages is not overwhelming. This supports a narrower claim about improved task performance, not a general increase in novice programmers’ efficacy. In the citing sentence, metacognitive behaviors are attributed to a different citation [70], not this source. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Classes of recursively enumerable sets and their decision problems
Rice, H. G. (1953) · Transactions of the American Mathematical Society · doi:10.1090/s0002-9947-1953-0053041-6 · Cited by 869 · see citing works
Designing the user interface: Strategies for effective human-computer interaction
Sriram, D. (1987) · Artificial Intelligence in Engineering · doi:10.1016/0954-1810(87)90189-0 · Cited by 758 · see citing works
Fisher, A. L., & Margolis, J. (2002) · ACM SIGCSE Bulletin · doi:10.1145/543812.543836 · Cited by 508 · see citing works
Enhancing the explanatory power of usability heuristics
Nielsen, J. (1994) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/259963.260333 · Cited by 335 · see citing works
Lidwell, W. M., Holden, K. L., & Butler, J. (2003) · Rockport eBooks · Cited by 313 · see citing works
An Empirical Investigation into Programming Language Syntax
Stefik, A., & Siebert, S. (2013) · ACM Transactions on Computing Education · doi:10.1145/2534973 · Cited by 295 · see citing works
What would other programmers do
Hartmann, B., MacDougall, D., Brandt, J., & Klemmer, S. (2010) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753478 · Cited by 280 · see citing works
Becker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372508 · Cited by 199 · see citing works
Self-citationA metric for software readability
Buse, R. P. L., & Weimer, W. R. (2008) · International Symposium on Software Testing and Analysis · doi:10.1145/1390630.1390647 · Cited by 182 · see citing works
Programmers' build errors: a case study (at google)
Seo, H., Sadowski, C., Elbaum, S. G., Aftandilian, E. E., & Bowdidge, R. W. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568255 · Cited by 180 · see citing works
Enhancing syntax error messages appears ineffectual
Denny, P. C., Luxton-Reilly, A., & Carpenter, D. (2014) · doi:10.1145/2591708.2591748 · Cited by 148 · see citing works
Self-citationNon-Native English Speakers Learning Computer Programming
Guo, P. J. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173970 · Cited by 145 · see citing works
All syntax errors are not equal
Denny, P., Luxton-Reilly, A., & Tempero, E. (2012) · Proceedings of the 17th ACM annual conference on Innovation and technology in computer science education · doi:10.1145/2325296.2325318 · Cited by 144 · see citing works
Self-citationUnderstanding the syntax barrier for novices
Denny, P., Luxton-Reilly, A., Tempero, E., & Hendrickx, J. (2011) · Proceedings of the 16th annual joint conference on Innovation and technology in computer science education · doi:10.1145/1999747.1999807 · Cited by 133 · see citing works
Self-citationOn Compiler Error Messages: What TheySayand What TheyMean
Traver, V. J. (2010) · Advances in Human-Computer Interaction · doi:10.1155/2010/602570 · Cited by 128 · see citing works
Personifying programming tool feedback improves novice programmers' learning
Lee, M. J., & Ko, A. J. (2011) · Proceedings of the seventh international workshop on Computing education research · doi:10.1145/2016911.2016934 · Cited by 125 · see citing works
An Effective Approach to Enhancing Compiler Error Messages
Becker, B. A. (2016) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2839509.2844584 · Cited by 125 · see citing works
Self-citationEffective compiler error message enhancement for novice programming students
Becker, B. A., Glanville, G., Iwashima, R., McDonnell, C., Goslin, K., & Mooney, C. (2016) · Computer Science Education · doi:10.1080/08993408.2016.1225464 · Cited by 101 · see citing works
Self-citationMeasuring the effectiveness of error messages designed for novice programmers
Marceau, G., Fisler, K., & Krishnamurthi, S. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953308 · Cited by 98 · see citing works
Do Developers Read Compiler Error Messages?
Barik, T., Smith, J., Lubick, K., Holmes, E., Feng, J., Murphy-Hill, E., & Parnin, C. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.59 · Cited by 86 · cited 2× · see citing works
“[6] provided compelling empirical evidence of the need for improving error”
BackgroundAhmed, U. Z., Kumar, P., Karkare, A., Kar, P., & Gulwani, S. (2018) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3183377.3183383 · Cited by 82 · see citing works
Seven deadly sins of introductory programming language design
McIver, L., & Conway, D. (2002) · International Conference on Software Engineering: Education and Practice (SEEP) · doi:10.1109/seep.1996.534015 · Cited by 78 · see citing works
What should we teach in an introductory programming course?
Gries, D. (1974) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/800183.810447 · Cited by 77 · see citing works
On Novices' Interaction with Compiler Error Messages: A Human Factors Approach
Prather, J., Pettit, R., McMurry, K. H., Peters, A., Homer, J., Simone, N., & Cohen, M. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106169 · Cited by 72 · see citing works
Self-citationAn introduction to object-oriented programming with a didactic microworld: objectKarel
Xinogalos, S., Satratzemi, M., & Dagdilelis, V. (2004) · Computers & Education · doi:10.1016/j.compedu.2004.09.005 · Cited by 68 · see citing works
Do Enhanced Compiler Error Messages Help Students?: Results Inconclusive.
Pettit, R. S., Homer, J., & Gee, R. (2017) · Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education · doi:10.1145/3017680.3017768 · Cited by 66 · cited 1× · see citing works
“Eforts focusing on enhancing/improving compiler error messages have been made by researchers such as Barik [4], Becker [13], Denny [19], Kohn [33], Pettit [45], Prather [46], and Karkare [2].”
BackgroundSelf-citationSolving the quandary between questionnaire length and response rate in educational research
Adams, L. L. M., & Gale, D. (1982) · Research in Higher Education · doi:10.1007/bf00976700 · Cited by 64 · see citing works
Designing computer system messages
Shneiderman, B. (1982) · Communications of the ACM · doi:10.1145/358628.358639 · Cited by 63 · see citing works
What Do CS1 Syllabi Reveal About Our Expectations of Introductory Programming Students?
Becker, B. A., & Fitzpatrick, T. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287485 · Cited by 63 · see citing works
Self-citationThe Effects of Enhanced Compiler Error Messages on a Syntax Error Debugging Test
Becker, B. A., Goslin, K., & Glanville, G. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159461 · Cited by 62 · see citing works
Self-citationAn exploration of novice compilation behaviour in BlueJ
Jadud, M. C. (2021) · Kent Academic Repository (University of Kent) · doi:10.22024/unikent/01.02.86458 · Cited by 61 · see citing works
Brown, P. J. (1983) · Communications of the ACM · doi:10.1145/2163.358083 · Cited by 57 · see citing works
Empowering students and building confidence in novice programmers through gauntlet
Flowers, T. R., Carver, C. A., & Jackson, J. (2005) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2004.1408551 · Cited by 56 · see citing works
Language Choice in Introductory Programming Courses at Australasian and UK Universities
Simon, Mason, R., Crick, T., Davenport, J. H., & Murphy, E. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159547 · Cited by 53 · see citing works
Error Message Readability and Novice Debugging Performance
Denny, P., Prather, J., & Becker, B. A. (2020) · Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3341525.3387384 · Cited by 50 · see citing works
Self-citationThe neglected battle fields of syntax errors
Kummerfeld, S., & Kay, J. (2003) · Cited by 42 · see citing works
The Error Behind The Message: Finding the Cause of Error Messages in Python
Kohn, T. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287381 · Cited by 41 · see citing works
PUFFT—The Purdue University fast FORTRAN translator
Rosen, S., Spurgeon, R. A., & Donnelly, J. K. (1965) · Communications of the ACM · doi:10.1145/365660.365671 · Cited by 40 · see citing works
How should compilers explain problems to developers?
Barik, T., Ford, D., Murphy-Hill, E. R., & Parnin, C. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236040 · Cited by 40 · see citing works
Redesigning introductory computer programming with HTML, JavaScript, and Java
Mahmoud, Q. H., Dobosiewicz, W., & Swayne, D. (2004) · ACM SIGCSE Bulletin · doi:10.1145/1028174.971344 · Cited by 37 · see citing works
Frequency distribution of error messages
Pritchard, D. (2015) · doi:10.1145/2846680.2846681 · Cited by 32 · see citing works
Study of novice programmers using Eclipse and Gild
Rigby, P. C., & Thompson, S. (2005) · doi:10.1145/1117696.1117718 · Cited by 24 · see citing works
Enhancing Python Compiler Error Messages via Stack
Thiselton, E., & Treude, C. (2019) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2019.8870155 · Cited by 24 · see citing works
An Exploration of the Effects of Enhanced Compiler Error Messages for Computer Programming Novices
Becker, B. A. (2015) · Arrow - TU Dublin (Technological University Dublin) · doi:10.13140/rg.2.2.26637.13288 · Cited by 22 · see citing works
Self-citationError messages are classifiers: a process to design and evaluate error messages
Wrenn, J., & Krishnamurthi, S. (2017) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3133850.3133862 · Cited by 21 · see citing works
Coull, N. (2008) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 19 · see citing works
‘My system gives excellent error messages’—or does it?
Brown, P. J. (1982) · Software Practice and Experience · doi:10.1002/spe.4380120110 · Cited by 18 · see citing works
Wexelblat, R. L. (1976) · doi:10.5555/800253.807695 · Cited by 17 · see citing works
Maglio, P. P., & Kandogan, E. (2004) · Queue · doi:10.1145/1036474.1036499 · Cited by 15 · see citing works
A methodology for objectively evaluating error messages
Isa, B. S., Boyle, J. M., Neal, A. S., & Simons, R. M. (1983) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/800045.801583 · Cited by 13 · see citing works
The Impact of "Cosmetic" Changes on the Usability of Error Messages
Dong, T., & Khandwala, K. (2019) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3312978 · Cited by 13 · see citing works
Syntax and Stack Overflow: A Methodology for Extracting a Corpus of Syntax Errors and Fixes
Wong, A. W., Salimi, A., Chowdhury, S., & Hindle, A. (2019) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2019.00048 · Cited by 12 · see citing works
Kurtz, T. E. (1978) · ACM SIGPLAN Notices · doi:10.1145/960118.808376 · Cited by 12 · see citing works
A Randomized Controlled Trial on the Wild Wild West of Scientific Computing with Student Learners
Rafalski, T. F., Uesbeck, P. M., Panks-Meloney, C., Daleiden, P., Allee, W., McNamara, A. A., & Stefik, A. (2019) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3291279.3339421 · Cited by 11 · see citing works
Towards Assessing the Readability of Programming Error Messages
Becker, B. A., Denny, P. C., Prather, J. E., Pettit, R., Nix, R., & Mooney, C. (2021) · Australasian Computing Education Conference · doi:10.1145/3441636.3442320 · Cited by 10 · see citing works
Self-citationKarvelas, I., Dillane, J., & Becker, B. A. (2020) · doi:10.1145/3416465.3416471 · Cited by 8 · see citing works
Self-citationError Messages as Rational Reconstructions.
Barik, T. (2018) · NCSU Libraries Repository (North Carolina State University Libraries) · Cited by 7 · see citing works
Better template error messages
Alexandrescu, A. (1999) · C/C++ users journal · Cited by 5 · see citing works
Novice programmer errors : analysis and diagnostics
McCall, D. (2016) · Kent Academic Repository (University of Kent) · Cited by 4 · see citing works
Barbero (2014) · doi:10.1016/j · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [29]29. James J Horning. 1976. What the Compiler Should Tell the User. In Compiler Construction: An Advanced Course, G Goos and J Hartmanis (Eds.). Springer- Verlag, Berlin-Heidelberg, 525-548.
- [35]35. Thomas Kurtz. 1978. BASIC. ACM SIGPLAN Notices -Special issue: History of programming languages conference 13, 8 (1978), 103-118. https://doi.org/10.1145/ 960118.808376
- [43]43. Cormac Murray. 2019. An Analysis of Programming Process Data in a CS1 Pro- gramming Module: Factors Infuencing Success. Masters Thesis. University College Dublin.
Cited by
Sorted by citations
Computing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316 · see citing works
“Recent work has attempted to put error messages into more natural language by focusing on readability, which has been shown to improve student understanding of error messages and the number of successful code corrections [12].”
BackgroundSelf-citationPrather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256 · see citing works
“Previous work shows that a system providing enhanced feedback specifically targeted at novice programmers can increase efficacy [34, 89] and metacognitive behaviors [87].”
BackgroundSelf-citationExploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145 · see citing works
Ericson, B. J., Denny, P. C., Prather, J. E., Duran, R., Hellas, A., Leinonen, J., Miller, C. S., Morrison, B. B., Pearce, J. L., & Rodger, S. H. (2022) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3571785.3574127 · Cited by 59 · see citing works
Self-citationCodeTailor: LLM-Powered Personalized Parsons Puzzles for Engaging Support While Learning Programming
Hou, X., Wu, Z., Wang, X., & Ericson, B. J. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3662032 · Cited by 51 · see citing works
“As an AI language model, I cannot”: Investigating LLM Denials of User Requests
Wester, J., Schrills, T., Pohl, H., & van Berkel, N. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642135 · Cited by 40 · see citing works
Combining LLM-Generated and Test-Based Feedback in a MOOC for Programming
Gabbay, H., & Cohen, A. (2024) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3657604.3662040 · Cited by 36 · see citing works
A Large Scale RCT on Effective Error Messages in CS1
Wang, S., Mitchell, J. C., & Piech, C. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630764 · Cited by 35 · see citing works
Pirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 37 · see citing works
Taylor, A., Vassar, A., Renzella, J., & Pearce, H. A. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630822 · Cited by 33 · see citing works
Lohr, D., Keuning, H., & Kiesler, N. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13107 · Cited by 33 · see citing works
Chat Overflow: Artificially Intelligent Models for Computing Education - renAIssance or apocAIypse?
Denny, P. C., Becker, B. A., Leinonen, J., & Prather, J. E. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588773 · Cited by 33 · see citing works
Self-citationExploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices
Xiao, R., Hou, X., & Stamper, J. C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650937 · Cited by 32 · see citing works
Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models
MacNeil, S., Denny, P. C., Tran, A., Leinonen, J., Bernstein, S., Hellas, A., Sarsa, S., & Kim, J. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636245 · Cited by 40 · see citing works
Self-citationChoi, S., & Kim, H. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-13107-x · Cited by 38 · see citing works
Enhancing Programming Error Messages in Real Time with Generative AI
Kimmel, B., Geisert, A. L., Yaro, L., Gipson, B., Hotchkiss, R., Osae-Asante, S. K., Vaught, H., Wininger, G., & Yamaguchi, C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3647967 · Cited by 29 · see citing works
Always Provide Context: The Effects of Code Context on Programming Error Message Enhancement
Santos, E. A., Prasad, P., & Becker, B. A. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617909 · Cited by 27 · see citing works
Self-citationMetacodenition: Scaffolding the Problem-Solving Process for Novice Programmers
Pechorina, Y., Anderson, K., & Denny, P. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576130 · Cited by 26 · see citing works
Self-citationNon-Expert Programmers in the Generative AI Future
Feldman, M. Q., & Anderson, C. J. (2024) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3663384.3663393 · Cited by 21 · see citing works
PythonPal: Enhancing Online Programming Education Through Chatbot-Driven Personalized Feedback
Palahan, S. (2025) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2025.3545084 · Cited by 18 · see citing works
Alami, A., & Ernst, N. (2025) · 2025 IEEE/ACM 18th International Conference on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1109/chase66643.2025.00016 · Cited by 15 · see citing works
Not the Silver Bullet: LLM-enhanced Programming Error Messages are Ineffective in Practice
Santos, E. A., & Becker, B. A. (2024) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3689535.3689554 · Cited by 15 · see citing works
Self-citationOn the use of feedback in learning computer programming by novices: a systematic literature mapping
Rocha, H. J. B., Tedesco, P. C. D. A. R., & Costa, E. D. B. (2022) · Informatics in Education · doi:10.15388/infedu.2023.09 · Cited by 13 · see citing works
Framework for SQL Error Message Design: A Data-Driven Approach
Taipalus, T., & Grahn, H. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3607180 · Cited by 11 · see citing works
Caton, S. J., Russell, S., & Becker, B. A. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3478431.3499419 · Cited by 10 · see citing works
Self-citationSympathy for the (Novice) Developer
Karvelas, I., & Becker, B. A. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3478431.3499347 · Cited by 9 · see citing works
Self-citationFirst Steps Towards Predicting the Readability of Programming Error Messages
Prather, J. E., Denny, P. C., Becker, B. A., Nix, R., Reeves, B. N., Randrianasolo, A. S., & Powell, G. (2023) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545945.3569791 · Cited by 8 · see citing works
Self-citationBrown, N. C. C., Weill-Tessier, P., Leinonen, J., Denny, P. C., & Kölling, M. (2025) · ACM Transactions on Computing Education · doi:10.1145/3737885 · Cited by 8 · see citing works
Self-citationOn How Novices Approach Programming Exercises Before and During Coding
Sáenz, J. P., & Russis, L. D. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3519655 · Cited by 6 · see citing works
Computing Education Research in the UK & Ireland
Becker, B. A., Bradley, S., Maguire, J., Black, M. M., Crick, T., Saqr, M., Sentance, S., & Quille, K. (2023) · Springer eBooks · doi:10.1007/978-3-031-25336-2_19 · Cited by 5 · see citing works
Self-citationSánchez‐Santillán, M., Fernandez‐Medina, C., Pérez, J. R. P., & Ruiz, M. P. P. (2023) · Computer Applications in Engineering Education · doi:10.1002/cae.22640 · Cited by 4 · see citing works
Sim, G., Lochrie, M., Zubair, M. S., Kerr, O., & Bates, M. I. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42280-5_19 · Cited by 4 · see citing works
How to Evaluate Blame for Gradual Types, Part 2
Lazarek, L., Greenman, B., Felleisen, M., & Dimoulas, C. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3607836 · Cited by 4 · see citing works
How Domain Experts Use an Embedded DSL
Rennels, L., & Chasins, S. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3622851 · Cited by 4 · see citing works
Keeping Mindful of Modality: A Comparison of Computer Science Education Resources for Learning
Johnson, M., Baker-Ramos, R., Hovey, C. L., & DiSalvo, B. J. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631819 · Cited by 4 · see citing works
Chapagain, J., Sajib, M. I., Prodan, R., & Rus, V. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72315-5_5 · Cited by 5 · see citing works
Teaching Programming Error Message Understanding
Bouvier, D. J., Lovellette, E., Santos, E. A., Becker, B. A., Dasigi, V. G., Forden, J., Glebova, O., Joshi, S., Kurkovsky, S., & Russell, S. (2024) · doi:10.1145/3598579.3689377 · Cited by 4 · see citing works
Self-citationDornbusch, M., & Vahrenhold, J. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671105 · Cited by 3 · see citing works
Error Messages for Students Taught Using a Systematic Program Design Curriculum
Dzhatdoyev, S., Rosiers, J. A. D., & Morazán, M. T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-74558-4_9 · Cited by 3 · see citing works
Evaluating Distance Measures for Program Repair
Koutcheme, C., Sarsa, S., Leinonen, J., Haaranen, L., & Hellas, A. (2023) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3568813.3600130 · Cited by 3 · see citing works
Solano, L. L., Renzella, J., & Vassar, A. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626253.3635483 · Cited by 3 · see citing works
Novice Perceptions on Effective Elements of PostgreSQL Error Messages
Taipalus, T., Grahn, H., Ritonummi, S., Siitonen, V., Vartiainen, T., & Zhidkikh, D. (2025) · ACM Transactions on Computing Education · doi:10.1145/3732790 · Cited by 3 · see citing works
Napa, G. M. B. (2023) · International Conference on Industrial Engineering and Operations Management (IEOM) · doi:10.46254/an13.20230628 · Cited by 2 · see citing works
Teaching Students To Use Programming Error Messages
Bouvier, D. J., Lovellette, E., Santos, E. A., Becker, B. A., Crick, T., Dasigi, V. G., Forden, J., Glebova, O., Joshi, S., Kurkovsky, S., et al. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3617650.3624950 · Cited by 2 · see citing works
Self-citationMtaho, A. B. (2023) · Bulletin of Social Informatics Theory and Application · doi:10.31763/businta.v7i2.649 · Cited by 2 · see citing works
Large-Scale Analysis of Error Frequencies in Logo Programming
Staub, J., & Chothia, Z. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3478431.3499393 · Cited by 2 · see citing works
How Do Learners With Varying Skills Perceive Misconception Indicators and Feedback?
Evans, A. C., & Lock, D. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729073 · Cited by 1 · see citing works
Designing a Supportive IDE to Help Novices Recognise and Recover from Programming Misconceptions
Evans, A. C., Liu, J., Wang, Z., & Zheng, M. (2022) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3501709.3544277 · Cited by 1 · see citing works
An Experiment on the Effects of Modularity on Code Modification and Understanding
Tempero, E. D., Blincoe, K., & Lottridge, D. M. (2023) · Australasian Computing Education Conference · doi:10.1145/3576123.3576138 · Cited by 1 · see citing works
Mind the Error Message: An Inverted Quiz Format to Direct Learner's Attention to Error Messages
Tsunoda, K., Masuhara, H., & Cong, Y. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588823 · Cited by 1 · see citing works
Portraits of Programmer Behavior in a Frame-Based Language
Dillane, J., Karvelas, I., & Becker, B. A. (2021) · doi:10.1145/3507923.3507933 · Cited by 1 · see citing works
Self-citationUma avaliação comparativa entre métricas de erro em um curso introdutório de programação com Python
Araujo, L. G. D. J., Bittencourt, R. A., & Chavez, C. F. G. (2022) · doi:10.5753/educomp.2022.19197
Programmers' Views on IDE Compilation Mechanisms
Karvelas, I., Dillane, J., & Becker, B. A. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617915
Self-citationOn Students’ Perception of Compiler Syntax Error Messages: A Human Factors Approach
Algaraibeh, S., Jeffery, C. L., & Soule, T. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-76815-6_1
A Design Recipe and Recipe-Based Errors for Regular Expressions
Morazán, M. T., Dzhatdoyev, S., Rosiers, J. A. D., Minić, T., Garced, A. M., & Fields, D. A. K. (2025) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.424.2
Santos, E. A., Salmon, A., & Hammer, K. (2025) · IEEE Access · doi:10.1109/access.2025.3613500
Exploring the quality and effectiveness of AI-generated feedback in introductory programming
Qian, Y., Liu, M., & Zhu, L. (2026) · Humanities and Social Sciences Communications · doi:10.1057/s41599-026-07920-7
Solano, L. L., Koutcheme, C., Leinonen, J., Vassar, A., & Renzella, J. (2026) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3770762.3772576
From Lambda to Ledger: An Architectural Comparison of Plinth and Plutarch
Oh, S., Liu, Z., & Wadler, P. L. (2026) · ACM SIGPLAN International Workshop on Functional Software Architecture (FUNARCH) · doi:10.1145/3830438.3830956
"It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2023) · arXiv
“Previous work shows that a system providing enhanced feedback specifically targeted at novice programmers can increase efficacy [34, 89] and metacognitive behaviors [87].”
BackgroundSelf-citation
- Authors
- Denny, P., Prather, J., Becker, B. A., Mooney, C., Homer, J., Albrecht, Z. C., & Powell, G. B.
- Affiliations
- University of Auckland (NZ)
- Abilene Christian University (US)
- University College Dublin (IE)
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-15
- OpenAlex
- W3163358064
- Cited by
- 60
- References
- 59
- Citations by year
- 2026: 3 · 2025: 9 · 2024: 18 · 2023: 21 · 2022: 8 · 2021: 1
- FWCI
- 13.94
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Research Repository UCD (University College Dublin) (publishedVersion, cc-by-nc-nd)
- Funders
- University of Auckland
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- readability, novice programmers, message length, sentence structure, vocabulary, readability metrics, human-computer interaction, programming education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
