The Error Behind The Message: Finding the Cause of Error Messages in Python
Abstract
The interaction between a novice programmer, and the compiler plays a crucial role in the learning process of the novice programmer. Of particular importance is the compiler's feedback on errors in the program code. Accordingly, compiler error messages are an important and active field of research. Yet, a language that has largely been left out of this discussion so far is Python. We have collected Python programs from high school students taking introductory courses. For each collected erroneous program, we sought to classify the effective error, and assess if the student was able to fix the error. Our study is a precursor to providing improved error messages in Python, and assess their effectiveness. As such, we are eventually interested in finding ways to automatically determine the effective error, so as to base the displayed message on. From our data, we found that a considerable part of students' errors can be attributed to minor mistakes, which can easily be identified and corrected. However, beyond such minor mistakes, a proper error diagnosis might have to be based on a goal/plan analysis of the entire program. Likewise, proper assessment of whether an error has been fixed frequently requires more context than is provided by the program alone.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Novices often struggle with indentation.
“As novices often struggle with indentation [24,27]”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [24]
Cites
Altadmri, A., & Brown, N. C. C. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677258 · Cited by 261 · see citing works
Identifying and correcting Java programming errors for introductory computer science students
Hristova, M., Misra, A., Rutter, M., & Mercuri, R. T. (2003) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/611892.611956 · Cited by 201 · see citing works
Enhancing syntax error messages appears ineffectual
Denny, P. C., Luxton-Reilly, A., & Carpenter, D. (2014) · doi:10.1145/2591708.2591748 · Cited by 148 · see citing works
All syntax errors are not equal
Denny, P., Luxton-Reilly, A., & Tempero, E. (2012) · Proceedings of the 17th ACM annual conference on Innovation and technology in computer science education · doi:10.1145/2325296.2325318 · Cited by 144 · see citing works
On Compiler Error Messages: What TheySayand What TheyMean
Traver, V. J. (2010) · Advances in Human-Computer Interaction · doi:10.1155/2010/602570 · Cited by 128 · see citing works
An Effective Approach to Enhancing Compiler Error Messages
Becker, B. A. (2016) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2839509.2844584 · Cited by 125 · see citing works
Identifying and correcting Java programming errors for introductory computer science students
Hristova, M., Misra, A., Rutter, M., & Mercuri, R. T. (2003) · ACM SIGCSE Bulletin · doi:10.1145/792548.611956 · Cited by 120 · see citing works
Meaningful categorisation of novice programmer errors
McCall, D., & Kölling, M. (2014) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2014.7044420 · Cited by 103 · see citing works
Identifying Top Java Errors for Novice Programmers
Jackson, J., Cobb, M. L., & Carver, C. A. (2006) · 2022 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2005.1611967 · Cited by 100 · see citing works
Measuring the effectiveness of error messages designed for novice programmers
Marceau, G., Fisler, K., & Krishnamurthi, S. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953308 · Cited by 98 · see citing works
Do Developers Read Compiler Error Messages?
Barik, T., Smith, J., Lubick, K., Holmes, E., Feng, J., Murphy-Hill, E., & Parnin, C. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.59 · Cited by 86 · cited 2× · see citing works
“More recent research has started to investigate if the displayed error messages actually correspond to the underlying error [8, 13], or if students read and react to error messages [2, 12, 16].”
BackgroundNienaltowski, M.-H., Pedroni, M., & Meyer, B. (2008) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1352135.1352192 · Cited by 78 · see citing works
On Novices' Interaction with Compiler Error Messages: A Human Factors Approach
Prather, J., Pettit, R., McMurry, K. H., Peters, A., Homer, J., Simone, N., & Cohen, M. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106169 · Cited by 72 · see citing works
The Effects of Enhanced Compiler Error Messages on a Syntax Error Debugging Test
Becker, B. A., Goslin, K., & Glanville, G. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159461 · Cited by 62 · see citing works
Schorsch, T. (1995) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/199688.199769 · Cited by 59 · see citing works
Brown, P. J. (1983) · Communications of the ACM · doi:10.1145/2163.358083 · Cited by 57 · see citing works
Automated feedback on programs means students need less help from teachers
Odekirk-Hash, E., & Zachary, J. L. (2001) · doi:10.1145/364447.364537 · Cited by 43 · see citing works
Frequency distribution of error messages
Pritchard, D. (2015) · doi:10.1145/2846680.2846681 · Cited by 32 · see citing works
A detector for non-literal Java errors
Dy, T. T., & Rodrigo, M. M. T. (2010) · Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930485 · Cited by 28 · see citing works
Kohn, T. (2017) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-a-010871088 · Cited by 15 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
On Designing Programming Error Messages for Novices: Readability and its Constituent Factors
Denny, P., Prather, J., Becker, B. A., Mooney, C., Homer, J., Albrecht, Z. C., & Powell, G. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445696 · Cited by 60 · see citing works
Error Message Readability and Novice Debugging Performance
Denny, P., Prather, J., & Becker, B. A. (2020) · Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3341525.3387384 · Cited by 50 · see citing works
Scaling Large Language Model-based Multi-Agent Collaboration
Qian, C., Xie, Z., Wang, Y., Liu, W., Zhu, K., Xia, H., Dang, Y., Du, Z., Chen, W., Yang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.07155 · Cited by 8 · see citing works
“This advantage can be attributed to random edge connections, which, in analogy to social networks, potentially link "un-acquainted" agents via a direct shortcut, making them adjacent "acquaintances" and implicitly reducing the average path length, thus resembling small-world properties.”
- Authors
- Kohn, T.
- Affiliations
- University of Cambridge (GB)
- Corresponding author
- Kohn, Tobias
- ORCID
- Kohn, Tobias
- Venue
- Proceedings of the 50th ACM Technical Symposium on Computer Science Education
- Publisher
- ACM
- Published
- 2019-02-22
- Pages
- 524-530
- OpenAlex
- W2915219867
- Cited by
- 41
- References
- 20
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 12 · 2023: 9 · 2022: 3 · 2021: 4 · 2020: 6 · 2019: 2
- FWCI
- 4.34
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, programming, and type systems
- Keywords
- compiler error messages, Python, novice programmers, error diagnosis, error classification, student programming errors, introductory programming, high school students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
