CoChat
Sign Up
Cited by 32Teaching and Learning ProgrammingSoftware Engineering ResearchSoftware Engineering Techniques and Practices

Correlates of programmer efficacy and their link to experience: a combined EEG and eye-tracking study

Open access
Peitek, Norman, Bergum, Annabelle, Rekrut, Maurice, Mucke, Jonas, Nadig, Matthias, Parnin, Chris, Siegmund, Janet, Apel, Sven·2022·Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
DOI 10.1145/3540250.3549084
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Despite similar education and background, programmers can exhibit vast differences in efficacy. While research has identified some potential factors, such as programming experience and domain knowledge, the effect of these factors on programmers' efficacy is not well understood. Aims: We aim at unraveling the relationship between efficacy (speed and correctness) and measures of programming experience. We further investigate the correlates of programmer efficacy in terms of reading behavior and cognitive load. Method: For this purpose, we conducted a controlled experiment with 37 participants using electroencephalography (EEG) and eye tracking. We asked participants to comprehend up to 32 Java source-code snippets and observed their eye gaze and neural correlates of cognitive load. We analyzed the correlation of participants' efficacy with popular programming experience measures. Results: We found that programmers with high efficacy read source code more targeted and with lower cognitive load. Commonly used experience levels do not predict programmer efficacy well, but self-estimation and indicators of learning eagerness are fairly accurate. Implications: The identified correlates of programmer efficacy can be used for future research and practice (e.g., hiring). Future research should also consider efficacy as a group sampling method, rather than using simple experience measures.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3540250.3549084

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 70

  1. Shulamyt Ajami, Yonatan Woodbridge, and Dror Feitelson. 2017. Syntax, Predi- cates, Idioms -What Really Affects Code Complexity?. In Proc. Int'l Conf. Program Comprehension (ICPC). Springer, 66ś76.
  2. Naser Al Madi, Cole Peterson, Bonita Sharif, and Jonathan Maletic. 2021. From Novice to Expert: Analysis of Token Level Effects in a Longitudinal Eye Tracking Study. In Proc. Int'l Conf. Program Comprehension (ICPC). IEEE, 172ś183.
  3. Salwa Aljehane, Bonita Sharif, and Jonathan Maletic. 2021. Determining Dif- ferences in Reading Behavior Between Experts and Novices by Investigating Eye Movement on Source Code Constructs During a Bug Fixing Task. In Proc. Symposium on Eye Tracking Research & Applications (ETRA). ACM, Article 30, 6 pages.
  4. Jennifer Bauer, Janet Siegmund, Norman Peitek, Johannes Hofmeister, and Sven Apel. 2019. Indentation: Simply a Matter of Style or Support for Program Com- prehension? In Proc. Int'l Conf. Program Comprehension (ICPC). ACM, 11.
  5. Roman Bednarik. 2012. Expertise-Dependent Visual Attention Strategies Develop over Time during Debugging with Multiple Code Representations. Int'l Journal of Human-Computer Studies 70, 2 (2012), 143ś155.
  6. Mahnaz Behroozi, Chris Parnin, and Titus Barik. 2019. Hiring Is Broken: What Do Developers Say About Technical Interviews?. In Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 1ś9.
  7. Mahnaz Behroozi, Shivani Shirolkar, Titus Barik, and Chris Parnin. 2020. Debug- ging Hiring: What Went Right and What Went Wrong in the Technical Interview Process. In Proc. Int'l Conf. Software Engineering: Software Engineering in Society (ICSE-SEIS). IEEE, 71ś80.
  8. Tanja Blascheck and Bonita Sharif. 2019. Visually Analyzing Eye Movements on Natural Language Texts and Source Code Snippets. In Proc. Symposium on Eye Tracking Research & Applications (ETRA). ACM, 14:1ś14:9.
  9. Jean-Marie Burkhardt, Françoise Détienne, and Susan Wiedenbeck. 2002. Object- Oriented Program Comprehension: Effect of Expertise, Task and Phase. Empirical Softw. Eng. 7, 2 (2002), 115ś156.
  10. Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye Movements in Code Reading: Relaxing the Linear Order. In Proc. Int'l Conf. Program Comprehension (ICPC). IEEE, 255ś265.
  11. João Castelhano, Isabel Duarte, Carlos Ferreira, João Durães, Henrique Madeira, and Miguel Castelo-Branco. 2019. The Role of the Insula in Intuitive Expert Bug Detection in Computer Code: An fMRI Study. Brain Imaging and Behavior 13, 3 (2019), 623ś637.
  12. Gary Charness, Uri Gneezy, and Michael Kuhn. 2012. Experimental Methods: Between-Subject and Within-Subject Design. Journal of Economic Behavior & Organization 81, 1 (2012), 1ś8.
  13. Igor Crk and Timothy Kluthe. 2014. Toward Using Alpha and Theta Brain Waves to Quantify Programmer Expertise. In Int'l Conf. Engineering in Medicine and Biology Society. IEEE, 5373ś5376.
  14. Martha Crosby, Jean Scholtz, and Susan Wiedenbeck. 2002. The Roles Beacons Play in Comprehension for Novice and Expert Programmers. In Annual Conf. Psychology of Programming Interest Group (PPIG). 58ś73.
  15. Martha Crosby and Jan Stelovsky. 1990. How Do We Read Algorithms? A Case Study. Computer 23, 1 (1990), 25ś35.
  16. Arnaud Delorme and Scott Makeig. 2004. EEGLAB: An Open Source Toolbox for Analysis of Single-Trial EEG Dynamics Including Independent Component Analysis. Journal of Neuroscience Methods 134, 1 (2004), 9ś21.
  17. Oscar Dieste, Alejandrina Aranda, Fernando Uyaguari, Burak Turhan, Ayse Tosun, Davide Fucci, Markku Oivo, and Natalia Juristo. 2017. Empirical Evaluation of the Effects of Experience on Code Quality and Programmer Productivity: An Exploratory Study. Empirical Softw. Eng. 22, 5 (2017), 2457ś2542.
  18. Anders Ericsson and Andreas Lehmann. 1996. Expert and Exceptional Perfor- mance: Evidence of Maximal Adaptation to Task Constraints. Annual Review of Psychology 47, 1 (1996), 273ś305.
  19. Marisa Exter, Secil Caskurlu, and Todd Fernandez. 2018. Comparing Computing Professionals' Perceptions of Importance of Skills and Knowledge on the Job and Coverage in Undergraduate Experiences. ACM Transactions on Computing Education (TOCE) 18, 4 (2018), 1ś29.
  20. Sarah Fakhoury, Yuzhan Ma, Venera Arnaoudova, and Olusola Adesope. 2018. The Effect of Poor Source Code Lexicon and Readability on Developers' Cognitive Load. In Proc. Int'l Conf. Program Comprehension (ICPC). IEEE, 286ś28610.
  21. Sarah Fakhoury, Devjeet Roy, Yuzhan Ma, Venera Arnaoudova, and Olusola Adesope. 2020. Measuring the Impact of Lexical and Structural Inconsistencies on Developers? Cognitive Load during Bug Localization. Empirical Softw. Eng. 3 (2020), 2140ś2178. Issue 25.
  22. Benjamin Floyd, Tyler Santander, and Westley Weimer. 2017. Decoding the Representation of Code in the Brain: An fMRI Study of Code Review and Expertise. In Proc. Int'l Conf. Software Engineering (ICSE). IEEE, 175ś186.
  23. David Gilmore and Thomas Green. 1988. Programming Plans and Programming Expertise. The Quarterly Journal of Experimental Psychology 40, 3 (1988), 423ś442.
  24. Wouter Groeneveld, Joost Vennekens, and Kris Aerts. 2021. Identifying Non- Technical Skill Gaps in Software Engineering Education: What Experts Expect But Students Don't Learn. ACM Transactions on Computing Education (TOCE) 22, 1 (2021), 1ś21.
  25. Roy Hessels, Diederick Niehorster, Chantal Kemner, and Ignace Hooge. 2017. Noise-Robust Fixation Detection in Eye Movement Data: Identification by Two- Means Clustering (I2MC). Behavior Research Methods 49, 5 (2017), 1802ś1823.
  26. Johannes Hofmeister, Janet Siegmund, and Daniel Holt. 2019. Shorter Identifier Names Take Longer to Comprehend. Empirical Softw. Eng. 24, 1 (2019), 417ś443.
  27. Anu Holm, Kristian Lukander, Jussi Korpela, Mikael Sallinen, and Kiti Müller. 2009. Estimating Brain Load from the EEG. TheScientificWorldJOURNAL 9 (2009), 639ś651.
  28. Toyomi Ishida and Hidetake Uwano. 2019. Synchronized analysis of eye move- ment and EEG during program comprehension. In Proc. Int'l Workshop on Eye Movements in Programming (EMIP). IEEE, 26ś32.
  29. Herbert Jasper. 1958. Report of the Committee on Methods of Clinical Examina- tion in Electroencephalography. Electroencephalogr. Clin. Neurophysiol. 10 (1958), 370ś375.
  30. Ahmad Jbara and Dror Feitelson. 2017. How Programmers Read Regular Code: A Controlled Experiment Using Eye Tracking. Empirical Softw. Eng. 22, 3 (2017), 1440ś1477.
  31. Magne Jorgensen, Gunnar Bergersen, and Knut Liestol. 2020. Relations Between Effort Estimates, Skill Indicators, and Measured Programming Skill. IEEE Trans. Softw. Eng. 47, 12 (2020), 2892ś2906.
  32. Aneta Kartali, Milica M. Janković, Ivan Gligorijević, Pavle Mijović, Bogdan Mi- jović, and Maria Chiara Leva. 2019. Real-Time Mental Workload Estimation Using EEG. In Human Mental Workload: Models and Applications. Springer International Publishing, 20ś34.
  33. Sebastian Kleinschmager and Stefan Hanenberg. 2011. How to Rate Programming Skills in Programming Experiments? A Preliminary, Exploratory, Study Based on University Marks, Pretests, and Self-Estimation. In Proc. Workshop on Evaluation and Usability of Programming Languages and Tools. ACM, 15ś24.
  34. Jürgen Koenemann and Scott Robertson. 1991. Expert Problem Solving Strategies for Program Comprehension. In Proc. Conf. Human Factors in Computing Systems (CHI). ACM, 125ś130.
  35. Makrina Kosti, Kostas Georgiadis, Dimitrios Adamos, Nikos Laskaris, Diomidis Spinellis, and Lefteris Angelis. 2018. Towards an Affordable Brain Computer Interface for the Assessment of Programmers' Mental Workload. Int.ĺ J. Human- Computer Studies 115 (2018), 52ś66.
  36. Thomas LaToza, David Garlan, James Herbsleb, and Brad Myers. 2007. Pro- gram Comprehension as Fact Finding. In Proc. Europ. Software Engineering Conf./Foundations of Software Engineering (ESEC/FSE). ACM, 361ś370.
  37. Seolhwa Lee, Andrew Matteson, Danial Hooshyar, SongHyun Kim, JaeBum Jung, GiChun Nam, and Heuiseok Lim. 2016. Comparing Programming Language Comprehension between Novice and Expert Programmers Using EEG Analysis. In Proc. Int'l Conf. on Bioinformatics and Bioengineering (BIBE). IEEE, 350ś355.
  38. Paul Luo Li, Amy Ko, and Jiamin Zhu. 2015. What Makes a Great Software Engineer?. In Proc. Int'l Conf. Software Engineering (ICSE), Vol. 1. IEEE, 700ś710.
  39. Rensis Likert. 1932. A Technique for the Measurement of Attitudes. Archives of Psychology 22, 140 (1932), 1ś55.
  40. Yu-Tzu Lin, Yi-Zhi Liao, Xiao Hu, and Cheng-Chih Wu. 2021. EEG Activities During Program Comprehension: An Exploration of Cognition. IEEE Access 9 (2021), 120407ś120421.
  41. Steve McConnell. 2011. What Does 10x Mean? Measuring Variations in Program- mer Productivity. In Making Software. O'Reilly & Associates, Inc., 567ś574.
  42. Jerry Mead, Simon Gray, John Hamer, Richard James, Juha Sorva, Caroline Clair, and Lynda Thomas. 2006. A Cognitive Approach to Identifying Measurable Milestones for Programming Skill Acquisition. ACM SIGCSE Bulletin 38, 4 (2006), 182ś194.
  43. Julio Medeiros, Ricardo Couceiro, João Castelhano, Castelo Branco, Gonçalo Duarte, Catarina Duarte, João Durães, Henrique Madeira, Paulo Carvalho, and César Teixeira. 2019. Software Code Complexity Assessment Using EEG Features. In Proc. Int'l Conf. Engineering in Medicine and Biology Society. IEEE, 1413ś1416.
  44. Júlio Medeiros, Ricardo Couceiro, Gonçalo Duarte, João Durães, João Castelhano, Catarina Duarte, Miguel Castelo-Branco, Henrique Madeira, Paulo de Carvalho, and César Teixeira. 2021. Can EEG Be Adopted as a Neuroscience Reference for Assessing Software Programmers' Cognitive Load? Sensors 21, 7 (2021), 2338.
  45. Takao Nakagawa, Yasutaka Kamei, Hidetake Uwano, Akito Monden, Kenichi Mat- sumoto, and Daniel German. 2014. Quantifying Programmers' Mental Workload During Program Comprehension Based on Cerebral Blood Flow Measurement: A Controlled Experiment. In Proc. Int'l Conf. Software Engineering (ICSE). ACM, 448ś451.
  46. Markus Nivala, Florian Hauser, Jürgen Mottok, and Hans Gruber. 2016. Develop- ing Visual Expertise in Software Engineering: An Eye Tracking Study. In Global Engineering Education Conference (EDUCON). IEEE, 613ś620.
  47. Francisco Ortin, Oscar Rodriguez-Prieto, Nicolas Pascual, and Miguel Garcia. 2020. Heterogeneous Tree Structure Classification to Label Java Programmers According to Their Expertise Level. Future Generation Computer Systems 105 (2020), 380ś394.
  48. Patrick Peachock, Nicholas Iovino, and Bonita Sharif. 2017. Investigating Eye Movements in Natural Language and C++ Source Code -A Replication Experi- ment. In Augmented Cognition. Neurocognition and Machine Learning. Springer, 206ś218.
  49. Jonathan Peirce, Jeremy Gray, Sol Simpson, Michael MacAskill, Richard Höchen- berger, Hiroyuki Sogo, Erik Kastman, and Jonas Lindelùv. 2019. PsychoPy2: Experiments in Behavior Made Easy. Behavior Research Methods 51, 1 (2019), 195ś203.
  50. Norman Peitek, Sven Apel, Chris Parnin, André Brechmann, and Janet Siegmund. 2021. Program Comprehension and Code Complexity Metrics: An fMRI Study. In Proc. Int'l Conf. Software Engineering (ICSE). ACM, 524ś536.
  51. Norman Peitek, Janet Siegmund, and Sven Apel. 2020. What Drives the Reading Order of Programmers? An Eye Tracking Study. In Proc. Int'l Conf. Program Comprehension (ICPC). ACM, 342ś353.
  52. Norman Peitek, Janet Siegmund, Chris Parnin, Sven Apel, and André Brechmann. 2018. Toward Conjoint Analysis of Simultaneous Eye-Tracking and fMRI Data for Program-Comprehension Studies. In Proc. Int'l Workshop on Eye Movements in Programming (EMIP). ACM, 1:1ś1:5.
  53. Maurice Rekrut, Mansi Sharma, Matthias Schmitt, Jan Alexandersson, and Anto- nio Krüger. 2020. Decoding Semantic Categories from EEG Activity in Object- Based Decision Tasks. In Proc. Int'l Winter Conf. Brain-Computer Interface (BCI). IEEE, 1ś7.
  54. Tobias Roehm, Rebecca Tiarks, Rainer Koschke, and Walid Maalej. 2012. How Do Professional Developers Comprehend Software?. In Proc. Int'l Conf. Software Engineering (ICSE). IEEE, 255ś265.
  55. James Shanteau. 1992. Competence in Experts: The Role of Task Characteristics. Organizational Behavior and Human Decision Processes 53, 2 (1992), 252ś266.
  56. James Shanteau and Thomas Stewart. 1992. Why Study Expert Decision Making? Some Historical Perspectives and Comments. Organizational Behavior and Human Decision Processes 53, 2 (1992), 95 ś 106.
  57. Zohreh Sharafi, Zéphyrin Soh, and Yann-Gaël Guéhéneuc. 2015. A System- atic Literature Review on the Usage of Eye-Tracking in Software Engineering. Information and Software Technology 67 (2015), 79ś107.
  58. Bonita Sharif, Michael Falcone, and Jonathan Maletic. 2012. An Eye-Tracking Study on the Role of Scan Time in Finding Source Code Defects. In Proc. Sympo- sium on Eye Tracking Research & Applications (ETRA). ACM, 381ś384.
  59. Ben Shneiderman and Richard Mayer. 1979. Syntactic/Semantic Interactions in Programmer Behavior: A Model and Experimental Results. Int'l J. Parallel Programming 8, 3 (1979), 219ś238.
  60. Janet Siegmund. 2016. Program Comprehension: Past, Present, and Future. In Int'l Conf. Software Analysis, Evolution, and Reengineering (SANER). IEEE, 13ś20.
  61. Janet Siegmund, Christian Kästner, Jörg Liebig, Sven Apel, and Stefan Hanenberg. 2014. Measuring and Modeling Programming Experience. Empirical Softw. Eng. 19, 5 (2014), 1299ś1334.
  62. Janet Siegmund, Norman Peitek, Chris Parnin, Sven Apel, Johannes Hofmeister, Christian Kästner, Andrew Begel, Anja Bethmann, and André Brechmann. 2017. Measuring Neural Efficiency of Program Comprehension. In Proc. Europ. Software Engineering Conf./Foundations of Software Engineering (ESEC/FSE). ACM, 140ś 150.
  63. Janet Siegmund and Jana Schumann. 2015. Confounding Parameters on Program Comprehension: A Literature Survey. Empirical Softw. Eng. 20, 4 (2015), 1159ś 1192.
  64. Elliot Soloway and Kate Ehrlich. 1984. Empirical Studies of Programming Knowl- edge. IEEE Trans. Softw. Eng. 10, 5 (1984), 595ś609.
  65. Harold Stanislaw, Beryl Hesketh, Sylvia Kanavaros, Tim Hesketh, and Ken Robin- son. 1994. A Note on the Quantification of Computer Programming Skill. Inter- national Journal of Human-Computer Studies 41, 3 (1994), 351ś362.
  66. Marie Vans, Anneliese von Mayrhauser, and Gabriel Somlo. 1999. Program Understanding Behavior During Corrective Maintenance of Large-Scale Software. Int. 'l Journal of Human-Computer Studies 51, 1 (1999), 31ś70.
  67. Susan Wiedenbeck. 1985. Novice/Expert Differences in Programming Skills. Int. 'l Journal of Man-Machine Studies 23, 4 (1985), 383ś390.
  68. Susan Wiedenbeck, Vikki Fix, and Jean Scholtz. 1993. Characteristics of the Mental Representations of Novice and Expert Programmers: An Empirical Study. Int. 'l Journal of Man-Machine Studies 39, 5 (1993), 793ś812.
  69. Martin Yeh, Dan Gopstein, Yu Yan, and Yanyan Zhuang. 2017. Detecting and Comparing Brain Activity in Short Program Comprehension Using EEG. In Frontiers in Education Conference. IEEE, 1ś5.
  70. Fanghui Zha and Yong Wang. 2021. Correlation Analysis of Subject Competition and Programming Ability for Novice Programmers. In Proc. Int'l Symp. System and Software Reliability (ISSSR). IEEE, 25ś31.

Cited by

Sorted by citations

Authors
Peitek, N., Bergum, A., Rekrut, M., Mucke, J., Nadig, M., Parnin, C., Siegmund, J., & Apel, S.
Affiliations
  • Saarland University (DE)
  • German Research Centre for Artificial Intelligence (DE)
  • Chemnitz University of Technology, Germany
  • North Carolina State University (US)
  • Chemnitz University of Technology (DE)
ORCID
Venue
Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering
Publisher
ACM
Published
2022-11-07
Pages
120-131
DOI
10.1145/3540250.3549084
OpenAlex
W4308643133
Cited by
32
References
4
Citations by year
2026: 9 · 2025: 7 · 2024: 13 · 2023: 3
FWCI
7.03
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
  • Publications of the UdS (Saarland University) (cc-by)
Funders
  • German Federal Ministry of Education and Research (01IS12050)
  • National Science Foundation (1755762)
  • European Research Council (101052182)
  • Deutsche Forschungsgemeinschaft (SI 2045/2-2, 389792660)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
programming experience, cognitive load, gaze, EEG, self-estimation, program comprehension, personnel selection, Java source code

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review