Fix the First, Ignore the Rest: Dealing with Multiple Compiler Error Messages
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
In order to help students learning to develop computer programs, several computing education researchers have analyzed the compiler error messages generated by novices' attempts to compile their programs. The goal is to help students diagnose the errors they make through the messages generated by the compiler. This paper builds on that previous work by applying a technique based on a heuristic well-known to programmers - fix the first error and ignore the rest - to the analysis of over 21 million compiler error messages from the Blackbox dataset. We find that the ranks and frequencies obtained by considering all error messages are generally consistent with previously published lists, but when we consider first messages only, these ranks and frequencies are different. These differences could have important implications for teaching, and can inform tool design and future research efforts.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
The BlueJ System and its Pedagogy
Kölling, M., Quig, B., Patterson, A., & Rosenberg, J. (2003) · Computer Science Education · doi:10.1076/csed.13.4.249.17496 · Cited by 484
37 Million Compilations: Investigating Novice Programming Mistakes in Large-Scale Student Data
Altadmri, A., & Brown, N. C. C. (2015) · Proceedings of the 46th ACM Technical Symposium on Computer Science Education · doi:10.1145/2676723.2677258 · Cited by 261
Identifying and correcting Java programming errors for introductory computer science students
Hristova, M., Misra, A., Rutter, M., & Mercuri, R. (2003) · Proceedings of the 34th SIGCSE technical symposium on Computer science education · doi:10.1145/611892.611956 · Cited by 201 · cited 2×
“Several authors have analyzed student understanding of error messages, developed tools that aempt to provide beer compiler error messages, and/or supplemented standard messages with additional information [7, 13, 16, 18, 19, 24, 28, 32].”
BackgroundAn analysis of patterns of debugging among novice computer science students
Ahmadzadeh, M., Elliman, D., & Higgins, C. (2005) · Proceedings of the 10th annual SIGCSE conference on Innovation and technology in computer science education · doi:10.1145/1067445.1067472 · Cited by 185
Blackbox: a large scale repository of novice programmers' activity
Brown, N. C. C., Kölling, M., McCall, D., & Utting, I. (2014) · Proceedings of the 45th ACM technical symposium on Computer science education · doi:10.1145/2538862.2538924 · Cited by 179 · cited 3×
“e list of most-frequent rst messages is notably dierent from the corresponding list for all messages, and from similar lists that have been reported before [11, 16, 20].”
BackgroundMethodEnhancing syntax error messages appears ineffectual
Denny, P., Luxton-Reilly, A., & Carpenter, D. (2014) · Proceedings of the 2014 conference on Innovation & technology in computer science education - ITiCSE '14 · doi:10.1145/2591708.2591748 · Cited by 148 · cited 2×
“Several authors have analyzed student understanding of error messages, developed tools that aempt to provide beer compiler error messages, and/or supplemented standard messages with additional information [7, 13, 16, 18, 19, 24, 28, 32].”
BackgroundAll syntax errors are not equal
Denny, P., Luxton-Reilly, A., & Tempero, E. (2012) · Proceedings of the 17th ACM annual conference on Innovation and technology in computer science education · doi:10.1145/2325296.2325318 · Cited by 144 · cited 2×
“[14] collected errors from student exercises using a custom “drill and practice” tool.”
BackgroundMethodAn Effective Approach to Enhancing Compiler Error Messages
Becker, B. A. (2016) · Proceedings of the 47th ACM Technical Symposium on Computing Science Education · doi:10.1145/2839509.2844584 · Cited by 125 · cited 1×
“[4, 5, 7] used an editor designed to collect and enhance error messages generated by the standard JDK compiler.”
MethodSelf-citationEffective compiler error message enhancement for novice programming students
Becker, B. A., Glanville, G., Iwashima, R., McDonnell, C., Goslin, K., & Mooney, C. (2016) · Computer Science Education · doi:10.1080/08993408.2016.1225464 · Cited by 101 · cited 3×
“[4, 5, 7] used an editor designed to collect and enhance error messages generated by the standard JDK compiler.”
BackgroundMethodSelf-citationNovice Java Programming Mistakes: Large-Scale Data vs. Educator Beliefs
Brown, N. C. C., & Altadmri, A. (2017) · ACM Transactions on Computing Education · doi:10.1145/2994154 · Cited by 100 · cited 4×
“As an example of how such lists can have long lifetimes in informing other research, in 2017 Brown and Altadmri [10] used a set of 18 errors derived from the 20 originally presented in 2003 by Hristova et al.”
BackgroundMethodInfluentialMeasuring the effectiveness of error messages designed for novice programmers
Marceau, G., Fisler, K., & Krishnamurthi, S. (2011) · Proceedings of the 42nd ACM technical symposium on Computer science education · doi:10.1145/1953163.1953308 · Cited by 98 · cited 1×
“Several authors have analyzed student understanding of error messages, developed tools that aempt to provide beer compiler error messages, and/or supplemented standard messages with additional information [7, 13, 16, 18, 19, 24, 28, 32].”
BackgroundCodeWrite: supporting student-driven practice of java
Denny, P., Luxton-Reilly, A., Tempero, E., & Hendrickx, J. (2011) · Proceedings of the 42nd ACM technical symposium on Computer science education · doi:10.1145/1953163.1953299 · Cited by 88 · cited 1×
“Compiler error messages have been studied for several decades [4], are essential for debugging, and can present barriers to student success [7, 13, 15].”
BackgroundDo Developers Read Compiler Error Messages?
Barik, T., Smith, J., Lubick, K., Holmes, E., Feng, J., Murphy-Hill, E., & Parnin, C. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.59 · Cited by 86
A New Metric to Quantify Repeated Compiler Errors for Novice Programmers
Becker, B. A. (2016) · Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/2899415.2899463 · Cited by 81 · cited 1×
“Other authors have used student error information to examine facets of student compilation or debugging behavior in general, including [1, 2, 6, 10, 12, 20].”
BackgroundSelf-citationOn Novices' Interaction with Compiler Error Messages: A Human Factors Approach
Prather, J., Pettit, R., McMurry, K. H., Peters, A., Homer, J., Simone, N., & Cohen, M. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106169 · Cited by 72 · cited 1×
“Several authors have analyzed student understanding of error messages, developed tools that aempt to provide beer compiler error messages, and/or supplemented standard messages with additional information [7, 13, 16, 18, 19, 24, 28, 32].”
BackgroundAn exploration of novice compilation behaviour in BlueJ
Jadud, M. C. (2021) · Kent Academic Repository (University of Kent) · doi:10.22024/unikent/01.02.86458 · Cited by 61
CAP: an automated self-assessment tool to check Pascal programs for syntax, logic and style errors
Schorsch, T. (1995) · Proceedings of the twenty-sixth SIGCSE technical symposium on Computer science education · doi:10.1145/199688.199769 · Cited by 59 · cited 2×
“Some novice programming environments, such as CAP [30] and BlueJ [23], have forced students to focus by only displaying the rst message.”
BackgroundEmpowering students and building confidence in novice programmers through gauntlet
Flowers, T., Carver, C. A., & Jackson, J. (2005) · 34th Annual Frontiers in Education, 2004. FIE 2004. · doi:10.1109/fie.2004.1408551 · Cited by 56 · cited 2×
“[18] chose nine common errors based on their experiences with students.”
BackgroundAggregate Compilation Behavior
Jadud, M. C., & Dorn, B. (2015) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2787622.2787718 · Cited by 41 · cited 1×
“Most papers so far have considered errors and error frequency [2, 10, 21, 29].”
BackgroundTeaching Object Orientation with the Blue Environment
Kölling, M. (1999) · Kent Academic Repository (University of Kent) · Cited by 36
Frequency distribution of error messages
Pritchard, D. (2015) · Proceedings of the 6th Workshop on Evaluation and Usability of Programming Languages and Tools · doi:10.1145/2846680.2846681 · Cited by 32 · cited 1×
“Most papers so far have considered errors and error frequency [2, 10, 21, 29].”
BackgroundAnalyses of Student Programming Errors In Java Programming Courses
Tuugalei, I., & Mow, C. (2012) · Cited by 30
A detector for non-literal Java errors
Dy, T., & Rodrigo, M. M. (2010) · Proceedings of the 10th Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930485 · Cited by 28 · cited 2×
“e list of most-frequent rst messages is notably dierent from the corresponding list for all messages, and from similar lists that have been reported before [11, 16, 20].”
BackgroundAnalyzing novice programmers' response to compiler error messages
Munson, J., & Schilling, E. A. (2016) · Journal of computing sciences in colleges · doi:10.5555/2835377.2835386 · Cited by 25
An Exploration of the Effects of Enhanced Compiler Error Messages for Computer Programming Novices
Becker, B. A. (2015) · Arrow - TU Dublin (Technological University Dublin) · doi:10.13140/rg.2.2.26637.13288 · Cited by 22
Self-citationWeb-scale data gathering with BlueJ
Utting, I., Brown, N. C. C., Kölling, M., McCall, D., & Stevens, P. (2012) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2361276.2361278 · Cited by 17 · cited 2×
“” e overarching contribution of this work is a large-scale analysis of the eects of isolating rst error messages from subsequent messages, using the Blackbox dataset [11, 33].”
BackgroundMethodFatal error in pass zero: how not to confuse novices
du Boulay, B., & MATTHEW, I. (1984) · Behaviour and Information Technology · doi:10.1080/01449298408901742 · Cited by 16
Smith, S. D., Zemljic, N., & Petersen, A. (2015) · doi:10.1145/2828959.2828980 · Cited by 8 · cited 1×
“One paper has discussed students’ use of GoTo-like control structures [31].”
BackgroundAnalyzing syntax error patterns among novice programmers
Moore, D., Parrish, A. S., & Cordes, D. W. (1997) · ACM Southeast Regional Conference (ACMSE) · doi:10.1145/2817460.2817508 · Cited by 4
Fincher, S. A. (2006) · doi:10.1007/978-0-230-20845-2 · Cited by 2
Research works printed but not linked: 2
- Murray Cormac, A Comparative Study of Java Compiler Error Profiles Using the Blackbox Dataset. Master's thesis
- Suzanne Marie Thompson. 2004. An Exploratory Study of Novice Programming Experiences and Errors. Master's thesis. Univ. of Victoria. Suzanne Marie Thompson. 2004. An Exploratory Study of Novice Programming Experiences and Errors. Master's thesis. Univ. of Victoria.
Cited by
Prather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
Self-citationBecker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372508 · Cited by 199
“…(Ben-Ari, 2007) [24], undecipherable (Traver, 2010) [202], intimidating (Hartz, 2012) [84], still very obviously less helpful than they could be (McCall & Kölling, 2015) [139], inscrutable (Ko, 2017) 1 , frustrating [again, 13 years after [72]] and a barrier to progress (Becker et al. , 2018) [21].”
BackgroundSelf-citation(2019) · doi:10.1145/3287324.3287374 · Cited by 104
Self-citationWhat Do CS1 Syllabi Reveal About Our Expectations of Introductory Programming Students?
Becker, B. A., & Fitzpatrick, T. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287485 · Cited by 63
Self-citationThe Effects of Enhanced Compiler Error Messages on a Syntax Error Debugging Test
Becker, B. A., Goslin, K., & Glanville, G. (2018) · Proceedings of the 49th ACM Technical Symposium on Computer Science Education · doi:10.1145/3159450.3159461 · Cited by 61
Self-citation
Metadata
- Authors
- Becker, B. A., Murray, C., Tao, T., Song, C., McCartney, R., & Sanders, K.
- Affiliations
- University College Dublin (IE)
- Fudan University (CN)
- University of Connecticut (US)
- Rhode Island College (US)
- Venue
- Proceedings of the 49th ACM Technical Symposium on Computer Science Education
- Publisher
- ACM
- Published
- 2018-02-21
- Pages
- 634-639
- OpenAlex
- W2791265653
- Cited by
- 61
- References
- 30
- Citations by year
- 2025: 12 · 2024: 7 · 2023: 6 · 2022: 6 · 2021: 12 · 2020: 6 · 2019: 8 · 2018: 4
- FWCI
- 8.49
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion)
- Funders
- University of Kent
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Logic, programming, and type systems
- Keywords
- compiler error messages, novice programmers, programming education, error diagnosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.