Do Developers Read Compiler Error Messages?
Abstract
In integrated development environments, developers receive compiler error messages through a variety of textual and visual mechanisms, such as popups and wavy red underlines. Although error messages are the primary means of communicating defects to developers, researchers have a limited understanding on how developers actually use these messages to resolve defects. To understand how developers use error messages, we conducted an eye tracking study with 56 participants from undergraduate and graduate software engineering courses at our university. The participants attempted to resolve common, yet problematic defects in a Java code base within the Eclipse development environment. We found that: 1) participants read error messages and the difficulty of reading these messages is comparable to the difficulty of reading source code, 2) difficulty reading error messages significantly predicts participants' task performance, and 3) participants allocate a substantial portion of their total task to reading error messages (13%-25%). The results of our study offer empirical justification for the need to improve compiler error messages for developers.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Novices read error messages and spend a substantial amount of programming time trying to understand them.
“Eye-tracking studies reveal that novices read error messages and spend a substantial amount of programming time trying to understand them [4].”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [4]
Cites
Eye movements in reading and information processing: 20 years of research
Rayner, K. (1998) · Psychological Bulletin · doi:10.1037/0033-2909.124.3.372 · Cited by 7,979 · see citing works
A note on a general definition of the coefficient of determination
NAGELKERKE, N. J. D. (1991) · Biometrika · doi:10.1093/biomet/78.3.691 · Cited by 5,927 · see citing works
An Overview of the Tesseract OCR Engine
Smith, R. W. (2007) · Proceedings of the International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2007.4376991 · Cited by 2,457 · see citing works
Response Time Effort: A New Measure of Examinee Motivation in Computer-Based Tests
Wise, S. L., & Kong, X. J. (2005) · Applied Measurement in Education · doi:10.1207/s15324818ame1802_2 · Cited by 617 · see citing works
Joyce, J. M. (2011) · International Encyclopedia of Statistical Science · doi:10.1007/978-3-642-04898-2_327 · Cited by 607 · see citing works
Why don't software developers use static analysis tools to find bugs?
Johnson, B., Song, Y., Murphy-Hill, E., & Bowdidge, R. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606613 · Cited by 568 · see citing works
Self-citationvan den Haak, M. J., de Jong, M. D. T., & Schellens, P. J. (2003) · Behaviour and Information Technology · doi:10.1080/0044929031000 · Cited by 487 · see citing works
Switching Languages, Switching Palabras (Words): An Electrophysiological Study of Code Switching
Moreno, E. M., Federmeier, K. D., & Marta Kutas, A. (2002) · Brain and Language · doi:10.1006/brln.2001.2588 · Cited by 307 · see citing works
Are Students Representatives of Professionals in Software Engineering Experiments?
Salman, I., Misirli, A. T., & Juristo, N. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.82 · Cited by 304 · see citing works
Why don’t software developers use static analysis tools to find bugs
Johnson, B., Song, Y., Murphy-Hill, E. R., & Bowdidge, R. W. (2013) · Cited by 268 · see citing works
Self-citationWhat developers want and need from program analysis: an empirical study
Christakis, M., & Bird, C. (2016) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/2970276.2970347 · Cited by 231 · see citing works
Improving automated source code summarization via an eye-tracking study of programmers
Rodeghero, P., McMillan, C., McBurney, P. W., Bosch, N., & D'Mello, S. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568247 · Cited by 198 · see citing works
Eye Movements in Code Reading: Relaxing the Linear Order
Busjahn, T., Bednarik, R., Begel, A., Crosby, M. E., Paterson, J. H., Schulte, C., Sharif, B., & Tamm, S. (2015) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2015.36 · Cited by 182 · see citing works
Programmers' build errors: a case study (at google)
Seo, H., Sadowski, C., Elbaum, S. G., Aftandilian, E. E., & Bowdidge, R. W. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568255 · Cited by 180 · see citing works
Analyzing individual performance of source code review using reviewers' eye movement
Uwano, H., Nakamura, M., Monden, A., & Matsumoto, K. (2006) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/1117309.1117357 · Cited by 177 · see citing works
Conscience without Cognition: The Effects of Subconscious Priming on Ethical Behavior
Welsh, D. T., & Ordóñez, L. D. (2013) · Academy of Management Journal · doi:10.5465/amj.2011.1009 · Cited by 175 · see citing works
Enhancing syntax error messages appears ineffectual
Denny, P. C., Luxton-Reilly, A., & Carpenter, D. (2014) · doi:10.1145/2591708.2591748 · Cited by 148 · see citing works
An eye-tracking study on the role of scan time in finding source code defects
Sharif, B., Falcone, M., & Maletic, J. I. (2012) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/2168556.2168642 · Cited by 144 · see citing works
How Programmers Debug, Revisited: An Information Foraging Theory Perspective
Lawrance, J., Bogart, C., Burnett, M., Bellamy, R., Rector, K., & Fleming, S. D. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.111 · Cited by 142 · see citing works
On Compiler Error Messages: What TheySayand What TheyMean
Traver, V. J. (2010) · Advances in Human-Computer Interaction · doi:10.1155/2010/602570 · Cited by 128 · see citing works
Eye Movements in Code Reading: Relaxing the Linear Order
Busjahn, T., Bednarik, R., Begel, A., Crosby, M. E., Paterson, J. H., Schulte, C., Sharif, B., & Tamm, S. (2016) · Cited by 111 · see citing works
Harris, L. R., & Jenkin, M. R. M. (2001) · Springer eBooks · doi:10.1007/978-0-387-21591-4_1 · Cited by 99 · see citing works
Measuring the effectiveness of error messages designed for novice programmers
Marceau, G., Fisler, K., & Krishnamurthi, S. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953308 · Cited by 98 · see citing works
Bednarik, R., & Tukiainen, M. (2008) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/1344471.1344497 · Cited by 57 · see citing works
Saccadic eye movements during reading
Jacobson, J. Z., & Dodwell, P. C. (1979) · Brain and Language · doi:10.1016/0093-934x(79)90058-0 · Cited by 46 · see citing works
Visual attention patterns during program debugging with an IDE
Hejmady, P., & Narayanan, N. H. (2012) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/2168556.2168592 · Cited by 46 · see citing works
Using Visual Momentum to Explain Disorientation in the Eclipse IDE
de Alwis, B., & Murphy, G. C. (2006) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2006.49 · Cited by 35 · see citing works
Using eye tracking to address limitations in think-aloud protocol
Cooke, L., & Cuddihy, E. (2005) · doi:10.1109/ipcc.2005.1494236 · Cited by 32 · see citing works
Frequency distribution of error messages
Pritchard, D. (2015) · doi:10.1145/2846680.2846681 · Cited by 32 · see citing works
Romero, P., Cox, R., du Boulay, B., & Lutz, R. (2002) · Lecture notes in computer science · doi:10.1007/3-540-46037-3_23 · Cited by 30 · see citing works
A cross-tool communication study on program analysis tool notifications
Johnson, B., Pandita, R., Smith, J., Ford, D., Elder, S., Murphy-Hill, E. R., Heckman, S. S., & Sadowski, C. (2016) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2950290.2950304 · Cited by 29 · see citing works
Self-citationBarik, T., Song, Y., Johnson, B., & Murphy-Hill, E. R. (2016) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2016.63 · Cited by 26 · see citing works
Self-citationFramework for measuring program comprehension
Siegmund, J. (2012) · Digitalen Hochschulbibliothek Sachsen-Anhalt (Universitäts- und Landesbibliothek Sachsen-Anhalt) · doi:10.25673/3877 · Cited by 11 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Using Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235 · see citing works
“Eye-tracking studies reveal that novices read error messages and spend a substantial amount of programming time trying to understand them [4].”
BackgroundBecker, B. A., Denny, P., Pettit, R., Bouchard, D., Bouvier, D. J., Harrington, B., Kamil, A., Karkare, A., McDonald, C., Osera, P.-M., et al. (2019) · Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3344429.3372508 · Cited by 199 · see citing works
“Rightfully so, students do not trust them [97] but importantly they do read them [12, 163].”
BackgroundA practical guide on conducting eye tracking studies in software engineering
Sharafi, Z., Sharif, B., Guéhéneuc, Y., Begel, A., Bednarik, R., & Crosby, M. E. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09829-4 · Cited by 133 · see citing works
“Do developers read error messages? (Barik et al., 2017)”
Metacognitive Difficulties Faced by Novice Programmers in Automated Assessment Tools
Prather, J., Pettit, R., McMurry, K., Peters, A., Homer, J., & Cohen, M. (2018) · Proceedings of the 2018 ACM Conference on International Computing Education Research · doi:10.1145/3230977.3230981 · Cited by 129 · see citing works
“…other recent studies utilize aspects of a human factors approach to an automated assessment tool such as the theroy behind error message design [60], design and personification of feedback [38], and eye-tracking to determine if novices read error messages and what they are reading when they do [3].”
MethodThe who, what, how of software engineering research: a socio-technical framework
Storey, M.-A., Ernst, N., Williams, C., & Kalliamvakou, E. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09858-z · Cited by 114 · see citing works
Confusion in Code Reviews: Reasons, Impacts, and Coping Strategies
Ebert, F., Castor, F., Novielli, N., & Serebrenik, A. (2019) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2019.8668024 · Cited by 85 · see citing works
“[74] aimed to obtain an empirical understanding of what makes a code change”
BackgroundSecurity Developer Studies with GitHub Users: Exploring a Convenience Sample
Acar, Y., Stransky, C., Wermke, D., Mazurek, M. L., & Fahl, S. (2017) · Symposium On Usable Privacy and Security · Cited by 73 · see citing works
On Novices' Interaction with Compiler Error Messages: A Human Factors Approach
Prather, J., Pettit, R., McMurry, K. H., Peters, A., Homer, J., Simone, N., & Cohen, M. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106169 · Cited by 72 · see citing works
“[2] use eye tracking software to examine whether intermediate students read error messages.”
Method"If you want, I can store the encrypted password"
Naiakshina, A., Danilova, A., Gerlitz, E., von Zezschwitz, E., & Smith, M. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300370 · Cited by 67 · see citing works
“Therefore, previous studies used convenience samples such as computer science students [2, 5, 21, 23] or GitHub developers [3, 15, 26, 38, 39].”
ResultProgramming Paradigms and Beyond
Krishnamurthi, S., & Fisler, K. (2019) · Cambridge University Press eBooks · doi:10.1017/9781108654555.014 · Cited by 65 · see citing works
“Getting students to actually read error messages may be difficult in practice (Denny, LuxtonReilly & Carpenter; 2104) (though this is not necessarily true of developers (Barik et al., 2017)).”
BackgroundThe Effects of Enhanced Compiler Error Messages on a Syntax Error Debugging Test
Becker, B. A., Goslin, K., & Glanville, G. (2018) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3159450.3159461 · Cited by 62 · see citing works
“Finally it should be noted that although this study and those reviewed in this section focus on novice programmers (with the exception of [1] who studied undergraduate and graduate students), there is evidence that compiler error messages also pose diculty for experienced developers [14, 19].”
BackgroundFix the First, Ignore the Rest: Dealing with Multiple Compiler Error Messages
Becker, B. A., Murray, C., Tao, T., Song, C., McCartney, R., & Sanders, K. (2018) · Proceedings of the 49th ACM Technical Symposium on Computer Science Education · doi:10.1145/3159450.3159453 · Cited by 61 · see citing works
On Designing Programming Error Messages for Novices: Readability and its Constituent Factors
Denny, P., Prather, J., Becker, B. A., Mooney, C., Homer, J., Albrecht, Z. C., & Powell, G. B. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445696 · Cited by 60 · see citing works
“[6] provided compelling empirical evidence of the need for improving error”
BackgroundAn exploratory study on confusion in code reviews
Ebert, F., Castor, F., Novielli, N., & Serebrenik, A. (2021) · Empirical Software Engineering · doi:10.1007/s10664-020-09909-5 · Cited by 54 · see citing works
“Barik et al. (2017) conducted an eye tracking study to understand how developers use compiler error messages.”
BackgroundError Message Readability and Novice Debugging Performance
Denny, P., Prather, J., & Becker, B. A. (2020) · Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3341525.3387384 · Cited by 50 · see citing works
“Even more striking, the difficulty of reading error messages was shown to be comparable to that of reading source code itself [3].”
BackgroundResultTesting probabilistic programming systems
Dutta, S., Legunsen, O., Huang, Z., & Misailovíc, S. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236057 · Cited by 50 · see citing works
Developer Reading Behavior While Summarizing Java Methods: Size and Context Matters
Abid, N. J., Sharif, B., Dragan, N., Alrasheed, H., & Maletic, J. I. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00052 · Cited by 50 · see citing works
“Eye Tracking Studies: Eye-tracking technology is being used in software engineering [20–25] to study how programmers read [4, 26–30], review [31–33], and summarize [17, 28, 34] source code.”
MethodOn Conducting Security Developer Studies with CS Students
Naiakshina, A., Danilova, A., Gerlitz, E., & Smith, M. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376791 · Cited by 48 · see citing works
“Both Jain et al. [21] and Barik et al. [10] argued to specifically have recruited students to avoid the unrealized bias of professionals being familiar with APIs.”
BackgroundBeyond the Code Itself: How Programmers Really Look at Pull Requests
Ford, D., Behroozi, M., Serebrenik, A., & Parnin, C. (2019) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) · doi:10.1109/icse-seis.2019.00014 · Cited by 46 · see citing works
Self-citationDoes stress impact technical interview performance?
Behroozi, M., Shirolkar, S., Barik, T., & Parnin, C. (2020) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3368089.3409712 · Cited by 42 · see citing works
“It is standard to simply compare lengths of fixations between different text [60], images [16, 82], and code [4, 14, 32] when understanding relative difficulty of tasks.”
BackgroundSelf-citationThe Error Behind The Message: Finding the Cause of Error Messages in Python
Kohn, T. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287381 · Cited by 41 · see citing works
“More recent research has started to investigate if the displayed error messages actually correspond to the underlying error [8, 13], or if students read and react to error messages [2, 12, 16].”
BackgroundHow should compilers explain problems to developers?
Barik, T., Ford, D., Murphy-Hill, E. R., & Parnin, C. (2018) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3236024.3236040 · Cited by 40 · see citing works
“[2] conducted an eye-tracking study with developers and found that they spent up to 25% of their task time on reading error messages.”
BackgroundSelf-citationListen to Developers! A Participatory Design Study on Security Warnings for Cryptographic APIs
Gorski, P. L., Acar, Y., Iacono, L. L., & Fahl, S. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376142 · Cited by 40 · see citing works
“al also found evidence that participants do read compiler error messages by conducting an eye-tracking study [10].”
BackgroundSynShine: Improved Fixing of Syntax Errors
Ahmed, T., Ledesma, N. R., & Devanbu, P. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3212635 · Cited by 38 · see citing works
Targeted Example Generation for Compilation Errors
Ahmed, U. Z., Sindhgatta, R., Srivastava, N., & Karkare, A. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00039 · Cited by 36 · see citing works
“have developed a large number of guidelines about how the error messages should be presented to developers [33], [34], [35], [36].”
BackgroundA Large Scale RCT on Effective Error Messages in CS1
Wang, S., Mitchell, J. C., & Piech, C. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630764 · Cited by 35 · see citing works
“Compiler-generated error messages are designed to explain a programming error to the programmer; however, many error messages are confusing for students who are learning to code [7, 12].”
BackgroundTahaei, M., Vaniea, K. E., Beznosov, K., & Wolters, M. K. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445616 · Cited by 35 · see citing works
“Focusing on usability issues, we know that a developers’ ability to read a compiler error message is correlated with their ability to complete a programming task correctly [9].”
Learning and programming challenges of rust
Zhu, S., Zhang, Z., Qin, B., Xiong, A., & Song, L. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510164 · Cited by 29 · see citing works
“Researchers performed several studies to understand how programmers interpret compiler error messages for traditional programming languages ( e.g. , C/C++) [4, 5].”
BackgroundDanilova, A., Naiakshina, A., & Smith, M. (2020) · Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering · doi:10.1145/3377811.3380387 · Cited by 25 · see citing works
“[12] conducted an eye-tracking study with student developers to investigate Java compiler error messages.”
Taipalus, T., Grahn, H., & Ghanbari, H. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111034 · Cited by 24 · see citing works
“In the context of programming languages, programmers use a considerable amount of time on reading and interpreting error messages (Barik et al., 2017).”
BackgroundPeitek, N., Siegmund, J., Parnin, C., Apel, S., & Brechmann, A. (2018) · doi:10.1145/3216723.3216725 · Cited by 21 · see citing works
“Similarly, much like Ogama allows researchers to define and store AOIs, we would need to describe complex eye-tracking events (e.g., fixation for at least half a second on a rich-identifier AOI).”
MethodInfluentialSelf-citationiTraceVis: Visualizing Eye Movement Data Within Eclipse
Clark, B., & Sharif, B. (2017) · IEEE Working Conference on Software Visualization (VISSOFT) · doi:10.1109/vissoft.2017.30 · Cited by 16 · see citing works
“Researchers have successfully used eye trackers to better understand how people read natural language [1], compiler errors [2], source code [3], [4], comprehend diagrams [5], and process 3D visualizations [6].”
BackgroundAddressing Compiler Errors: Stack Overflow or Large Language Models?Preprint
Widjojo, P., & Treude, C. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4529345 · Cited by 16 · see citing works
Charles, T., & Gwilliam, C. B. (2023) · Journal for STEM Education Research · doi:10.1007/s41979-022-00084-4 · Cited by 15 · see citing works
An ontology knowledge inspection methodology for quality assessment and continuous improvement
Roldán-Molina, G. R., Ruano-Ordás, D., Basto-Fernandes, V., & Méndez, J. R. (2021) · Data & Knowledge Engineering · doi:10.1016/j.datak.2021.101889 · Cited by 15 · see citing works
On Eye Tracking in Software Engineering
Grabinger, L., Häuser, F., Wolff, C. G., & Mottok, J. (2024) · SN Computer Science · doi:10.1007/s42979-024-03045-3 · Cited by 14 · see citing works
Henley, A. Z., Muçlu, K., Christakis, M., Fleming, S. D., & Bird, C. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173731 · Cited by 14 · see citing works
“A prior study using eye-tracking also observed programmers having difficulties reading automated analysis warnings [16], and other researchers proposed visualizations to improve comprehension [15].”
BackgroundGorski, P. L., Möller, S., Wiefling, S., & Iacono, L. L. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3094171 · Cited by 12 · see citing works
“Previous studies focused, e.g., on debugging and reading code [53], [54], [55], [56], and reading compiler error messages in the console [57].”
BackgroundFramework for SQL Error Message Design: A Data-Driven Approach
Taipalus, T., & Grahn, H. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3607180 · Cited by 11 · see citing works
“As developers use a signiicant amount of time reading and interpreting error messages [5], it seems natural that both the software industry and academia are interested in the qualities of error messages.”
BackgroundInfluentialTowards Assessing the Readability of Programming Error Messages
Becker, B. A., Denny, P. C., Prather, J. E., Pettit, R., Nix, R., & Mooney, C. (2021) · Australasian Computing Education Conference · doi:10.1145/3441636.3442320 · Cited by 10 · see citing works
“‘Improved’ or ‘enhanced’ error messages have been incorporated into many tools such as novice code editors and automated assessment tools [3, 4, 8, 16, 19, 28, 32, 33, 41], often with positive effects such as reducing the number of errors overall [5].”
BackgroundAssessing Workload Perception in Introductory Computer Science Projects using NASA-TLX
Madi, N. A., Peng, S., & Rogers, T. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3478431.3499406 · Cited by 11 · see citing works
“Furthermore, [3] used eye tracking during reading error messages and found that success in reading error messages is a significant predictor of successful debugging.”
BackgroundDesign, implementation and evaluation of the Hedy programming language
Gilsing, M., Pelay, J., & Hermans, F. (2022) · Journal of Computer Languages · doi:10.1016/j.cola.2022.101158 · Cited by 9 · see citing works
Mansoor, N., Peterson, C. S., Dodd, M. D., & Sharif, B. (2023) · ACM Transactions on Computing Education · doi:10.1145/3632530 · Cited by 9 · see citing works
“Titus et al. showed via an eye-tracking study that CS students found reading error messages equally hard compared to source code [ 7 ].”
Methodology Matters: How We Study Socio-Technical Aspects in Software EngineeringPreprint
Williams, C., Storey, M.-A. D., Ernst, N., Zagalsky, A., & Kalliamvakou, E. (2019) · arXiv (Cornell University) · Cited by 9 · see citing works
Ethical issues in empirical studies using student subjects: Re-visiting practices and perceptions
Liebel, G., & Chakraborty, S. (2021) · Empirical Software Engineering · doi:10.1007/s10664-021-09958-4 · Cited by 8 · see citing works
“Table 2 Primary studies reporting all of voluntariness, compensation, and status of ethics approval Key Study Type Sample Size Voluntariness Compensation Ethics Approval Anvari et al. (2017) Mixed-methods 75 voluntary Other Yes Tu et al. (2016) Controlled experiment 48 voluntary Movie vouchers Yes Dahan et al. (2014)…”
MethodUnderstanding Xacro Misunderstandings
Albergo, N., Rathi, V., & Ore, J. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9812349 · Cited by 8 · see citing works
Gradual Programming in Hedy: A First User Study
Gilsing, M., & Hermans, F. (2021) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc51201.2021.9576236 · Cited by 8 · see citing works
“This confirms earlier work on error messages, e.g. the work by Barik which showed that novice programmers read the error messages[28], but struggle with them more than experts [29].”
Accelerating OCR-Based Widget Localization for Test Automation of GUI Applications
Qian, J., Ma, Y., Lin, C., & Chen, L. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556966 · Cited by 8 · see citing works
“[16] use OCR to match file names for understanding compilation errors.”
MethodFirst Steps Towards Predicting the Readability of Programming Error Messages
Prather, J. E., Denny, P. C., Becker, B. A., Nix, R., Reeves, B. N., Randrianasolo, A. S., & Powell, G. (2023) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545945.3569791 · Cited by 8 · see citing works
“Barik et al. [3] indicated that programmers spend as much as 25% of total task time reading error messages.”
BackgroundBarriers in Front-End Web Development
Samudio, D. I., & LaToza, T. D. (2022) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc53370.2022.9833127 · Cited by 6 · see citing works
“…debugging easier by supporting following data dependencies backwards from erroneous output to the responsible code [17], making error messages easier to fix by offering actionable error messages [18], or making code easier to understand by reducing barriers to navigating call relationships [19].”
BackgroundA Mixed-Methods Study of Novice Programmer Interaction with Python Error Messages
D'souza, R., Bhayana, M., Ahmadzadeh, M., & Harrington, B. (2019) · Western Canadian Conference on Computing Education · doi:10.1145/3314994.3325090 · Cited by 6 · see citing works
“The ability to use such feedback effectively for debugging is crucial in the development of novice programmers [1, 7] but existing research has shown that current error messages (EMs) are inadequate for novices [3] and can act as a source of frustration [12].”
BackgroundDemystifying “bad” error messages in data science libraries
Tao, Y., Chen, Z., Liu, Y., Xuan, J., Xu, Z., & Qin, S. (2021) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3468264.3468560 · Cited by 6 · see citing works
“While previous work mostly addressed the quality of error messages generated by compilers [3][4][34], system software [36][39], and static analysis tools [2][20], limited knowledge is available about this issue in the domain of data science.”
BackgroundOrtloff, A.-M., Grohs, J. A., Lenau, S., & Smith, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714022 · Cited by 5 · see citing works
“This is exacerbated by the fact that truly random sampling is rare in the behavioural sciences [31], and especially for specialized populations like software developers, administrators, researchers, etc. [1, 7, 52].”
ChameleonIDE: Untangling Type Errors Through Interactive Visualization and Exploration
Fu, S., Dwyer, T., Stuckey, P. J., Wain, J., & Linossier, J. (2023) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc58990.2023.00029 · Cited by 5 · see citing works
“As put in [36], “Error messages appear to take the form of natural language, yet are as difficult to read as source code.”
BackgroundA framework for improving error messages in dependently-typed languages
Eremondi, J., Swierstra, W., & Hage, J. (2019) · Open Computer Science · doi:10.1515/comp-2019-0001 · Cited by 5 · see citing works
Mastering Variation in Human Studies
Siegmund, J., Peitek, N., Apel, S., & Siegmund, N. (2020) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3406544 · Cited by 5 · see citing works
“3 Same Results: Do Developers Read Compiler Error Messages? In the study by Barik and others, developers’ gaze behavior was observed with eye tracking [4].”
BackgroundMethodKeeping Mindful of Modality: A Comparison of Computer Science Education Resources for Learning
Johnson, M., Baker-Ramos, R., Hovey, C. L., & DiSalvo, B. J. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631819 · Cited by 4 · see citing works
“Compared to what is known about how programming error messages are interpreted [4, 5, 8, 33], it is surprising to see that even user-controlled text output is sometimes difficult to draw meaning from.”
BackgroundTeaching Programming Error Message Understanding
Bouvier, D. J., Lovellette, E., Santos, E. A., Becker, B. A., Dasigi, V. G., Forden, J., Glebova, O., Joshi, S., Kurkovsky, S., & Russell, S. (2024) · doi:10.1145/3598579.3689377 · Cited by 4 · see citing works
“This unfriendly output often hampers learning, resulting in students often spending up to a quarter of their time simply working through possible errors, a challenge as complex as reading source code itself [5].”
Bitcoin’s APIs in Open-Source Projects: Security Usability Evaluation
Tschannen, P., & Ahmed, A. (2020) · Electronics · doi:10.3390/electronics9071077 · Cited by 4 · see citing works
Developers' Perspective on Today's and Tomorrow's Programming Tool Assistance: A Survey
Kuang, P., Söderberg, E., & Höst, M. (2024) · International Conference on Art, Science, and Engineering of Programming (<Programming>) · doi:10.1145/3660829.3660848 · Cited by 3 · see citing works
“Since the understandability of analysis results stands out as a barrier to adoption in the literature [3, 4, 6], we designed questions specifically asking participants whether or not their English proficiency has an impact on their understanding of error messages or program analysis results.”
Building Blocks Towards More Effective SQL Error Messages
Taipalus, T., & Grahn, H. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653552 · Cited by 3 · see citing works
Dornbusch, M., & Vahrenhold, J. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671105 · Cited by 3 · see citing works
“Barik et al. highlighted the cognitive challenges programmers face when interpreting compiler error messages, equating their complexity to that of reading source code itself [1].”
Error Messages for Students Taught Using a Systematic Program Design Curriculum
Dzhatdoyev, S., Rosiers, J. A. D., & Morazán, M. T. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-74558-4_9 · Cited by 3 · see citing works
Solano, L. L., Renzella, J., & Vassar, A. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626253.3635483 · Cited by 3 · see citing works
“Unfortunately, the notoriously cryptic nature of PEMs makes them at best, largely ineffective, and at worst, actively detrimental, in assisting novices with debugging [1, 9].”
BackgroundNovice Perceptions on Effective Elements of PostgreSQL Error Messages
Taipalus, T., Grahn, H., Ritonummi, S., Siitonen, V., Vartiainen, T., & Zhidkikh, D. (2025) · ACM Transactions on Computing Education · doi:10.1145/3732790 · Cited by 3 · see citing works
“Software developers spend a significant amount of time interpreting error messages [4].”
Visual Attention Patterns in Finding Source Code Defects
Tablatin, C. L. S. (2023) · WSEAS Transactions on Information Science and Applications archive · doi:10.37394/23209.2023.20.40 · Cited by 2 · see citing works
The Impact of Compiler Warnings on Code Quality in C++ Projects
Johansson, A. A., Holmberg, C., Neto, F. G. D. O., & Leitner, P. (2024) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3643916.3644410 · Cited by 2 · see citing works
“Barik et al. [3] investigated whether developers even read error messages from compilers.”
BackgroundAn Analysis Scheme to Interpret Students' Cognitive Process in Error Finding Test
Liu, L., Liu, W., Li, X., Xu, J., & Cheng, W. (2020) · World Symposium on Software Engineering (WSSE) · doi:10.1145/3425329.3425350 · Cited by 2 · see citing works
“Among various physical measurement methods[1], eye tracking measurement is the most commonly used[9].”
BackgroundHow Do Programming Students Read and Act upon Compiler Error Messages?
Rodrigo, M. M. T., & Tablatin, C. L. S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35017-7_11 · Cited by 2 · see citing works
What Warnings Do Engineers Really Fix? The Compiler That Cried Wolf
Kudrjavets, G., Kumar, A., & Rastogi, A. (2023) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00068 · Cited by 1 · see citing works
“Benefit in this context means that warnings detect serious defects, have less false positives, and engineers trust and understand the diagnostic reports [6], [7].”
BackgroundEnhanced SQL error messages facilitate faster error fixing
Taipalus, T., Grahn, H., & Knutas, A. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10695-1 · Cited by 1 · see citing works
“Interpreting error messages is a significant aspect of the work undertaken by software devel - opers (Barik et al. 2017).”
Using eye tracking to identify features of peer parity on stack overflow
Ford, D. (2017) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2017.8103489 · Cited by 1 · see citing works
Notification Messages Considering Human Centered Design
Shirogane, J., Arizono, Y., Iwata, H., & Fukazawa, Y. (2018) · Smart innovation, systems and technologies · doi:10.1007/978-3-319-97679-2_6 · Cited by 1 · see citing works
An Empirical Study of Test-Driven Development vs. Test-Last Development Using Eye Tracking
Choma, J., Guerra, E., da Silva, T. S., Albuquerque, T., Albuquerque, V. G., & Zaina, L. A. M. (2019) · Communications in computer and information science · doi:10.1007/978-3-030-36701-5_2 · Cited by 1 · see citing works
Investigating Compilation Errors of Students Learning Haskell
Németh, B., Choi, E., Makihara, E., & Iida, H. (2019) · Electronic Proceedings in Theoretical Computer Science · doi:10.4204/eptcs.295.4
“A study with eye tracking supports that understanding error messages is vital for the success of a programming task and as difficult as reading source code [8].”
BackgroundUnderstanding Eye Gaze Patterns in Code Comprehension
Saddler, J. A. (2020) · Lincoln (University of Nebraska)
Representational Learning Approach for Predicting Developer Expertise Using Eye Movements
Maan, S. (2020) · Lincoln (University of Nebraska)
Programmers' Views on IDE Compilation Mechanisms
Karvelas, I., Dillane, J., & Becker, B. A. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617915
Detecting Compiler Error Recovery Defects via Program Mutation Exploration
Tang, Y., Zhang, J., Li, H., Huang, Z., & Jiang, H. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3510912 · Cited by 1 · see citing works
“…a research in a Java code base with the Eclipse development environment shows that, the difficulty of reading error recovery diagnostics is comparable to the difficulty of reading source code, and developers spend up to 25% of their programming time on reading error recovery diagnostics [20].”
A Parsing Technique for Enhancing Compiler Syntax Error Messages for Student Programmers
Algaraibeh, S., Jeffery, C. L., Soule, T., & Dousay, T. A. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893401
“Also, they found that as revisits to error messages increase, the probability of successfully resolving a compiler error decreases [5].”
JestViz: Towards Supporting Novice Web Developers Debug JavaScript
Shanbhag, S., & Chimalakonda, S. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-84391-4_8
Investigating the Understandability of Review Comments on Code Change Requests
Rahman, M. S., Codabux, Z., & Roy, C. K. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00087
“Uwano et al. [55] and Barik et al. [56] used eye-tracking to explore code understandability and error interpretation, respectively.”
Leveraging Error Information from Identical Learning Materials for Debugging Effectiveness
Takahashi, K. (2025) · International Journal of Computer Applications · doi:10.5120/ijca2025925263
Smith, M., Ortloff, A.-M., Lenau, S., & Grohs, J. A. (2025) · OSF Preprints (OSF Preprints) · doi:10.17605/osf.io/qy2u9
Python Programming Errors in if-Conditionals: Novices' Reactions to Error Messages
Bachmann, C., Kohn, T., Komm, D., & Loyo, A. H. (2026) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3830800.3830811
Almeida, C. D. P., Buenaño, N. D. L. J., Lucio, D. R., Ambato-Jordán, C., & Puruncajas, Á. J. M. (2026) · Buildings · doi:10.3390/buildings16193776
- Authors
- Barik, T., Smith, J., Lubick, K., Holmes, E., Feng, J., Murphy-Hill, E., & Parnin, C.
- Affiliations
- North Carolina State University (US)
- Washington and Lee University (US)
- Venue
- 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)
- Publisher
- IEEE
- Published
- 2017-05
- Pages
- 575-585
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W2617813225
- Cited by
- 86
- References
- 33
- Citations by year
- 2026: 2 · 2025: 8 · 2024: 10 · 2023: 12 · 2022: 7 · 2021: 10 · 2020: 12 · 2019: 13 · 2018: 8 · 2017: 4
- FWCI
- 7.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- compiler error messages, developer behavior, defect resolution, integrated development environment, eclipses, eye tracking, visual feedback, Java, task performance, source code reading
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
