A Survey on the Usage of Eye-Tracking in Computer Programming
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Traditional quantitative research methods of data collection in programming, such as questionnaires and interviews, are the most common approaches for researchers in this field. However, in recent years, eye-tracking has been on the rise as a new method of collecting evidence of visual attention and the cognitive process of programmers. Eye-tracking has been used by researchers in the field of programming to analyze and understand a variety of tasks such as comprehension and debugging. In this article, we will focus on reporting how experiments that used eye-trackers in programming research are conducted, and the information that can be collected from these experiments. In this mapping study, we identify and report on 63 studies, published between 1990 and June 2017, collected and gathered via manual search on digital libraries and databases related to computer science and computer engineering. Among the five main areas of research interest are program comprehension and debugging, which received an increased interest in recent years, non-code comprehension, collaborative programming, and requirements traceability research, which had the fewest number of publications due to possible limitations of the eye-tracking technology in this type of experiments. We find that most of the participants in these studies were students and faculty members from institutions of higher learning, and while they performed programming tasks on a range of programming languages and programming representations, we find Java language and Unified Modeling Language (UML) representation to be the most used materials. We also report on a range of eye-trackers and attention tracking tools that have been utilized, and find Tobii eye-trackers to be the most used devices by researchers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Eye movements in reading and information processing: 20 years of research
Rayner, K. (1998) · Psychological Bulletin · doi:10.1037/0033-2909.124.3.372 · Cited by 7,979
Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
Systematic Mapping Studies in Software Engineering
Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2008.8 · Cited by 3,239 · cited 2×
“We adopted the classification scheme used by [76], as it was cited and recommended by [51].”
MethodRayner, K. (2009) · Quarterly Journal of Experimental Psychology · doi:10.1080/17470210902816461 · Cited by 2,682
A systematic review of systematic review process research in software engineering
Kitchenham, B., & Brereton, P. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.07.010 · Cited by 1,094
How are Java software developers using the Eclipse IDE?
Murphy, G. C., Kersten, M., & Findlater, L. (2006) · IEEE Software · doi:10.1109/ms.2006.105 · Cited by 413 · cited 3×
“—[S JAP13] The software visualization tool examined, SeeIT 3D [44], is implemented as an Eclipse plugin.”
MethodInfluentialImproving automated source code summarization via an eye-tracking study of programmers
Rodeghero, P., McMillan, C., McBurney, P. W., Bosch, N., & D'Mello, S. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568247 · Cited by 198 · cited 1×
“—[62] and [61]: A more comprehensive analysis of the data from the same experiment was reported in [RLMM15].”
Background
Research works printed but not linked: 73
- 10.1007/s10664-014-9315-y · DOI
- 10.1080/10494820.2015.1073746 · DOI
- Roman Bednarik , Teresa Busjahn , and Carsten Schulte (Eds.). 2014 . Eye Movements in Programming Education: Analyzing the Experts Gaze. Technical Report . University of Eastern Finland , Joensuu, Finland . Roman Bednarik, Teresa Busjahn, and Carsten Schulte (Eds.). 2014. Eye Movements in Programming Education: Analyzing the Experts Gaze. Technical Report. University of Eastern Finland, Joensuu, Finland.
- 10.1109/VLHCC.2005.20 · doi:10.1109/vlhcc.2005.20
- Roman Bednarik , Niko Myller , Erkki Sutinen , and Markku Tukiainen . 2005 . Effects of experience on gaze behaviour during program animation . In Proceedings of the 17th Annual Psychology of Programming Interest Group Workshop. 49--61 . Roman Bednarik, Niko Myller, Erkki Sutinen, and Markku Tukiainen. 2005. Effects of experience on gaze behaviour during program animation. In Proceedings of the 17th Annual Psychology of Programming Interest Group Workshop. 49--61.
- Roman Bednarik , Niko Myller , Erkki Sutinen , and Markku Tukiainen . 2006 . Analyzing individual differences in program comprehension . Technology Instruction Cognition and Learning 3 , 3/4 (2006), 205 . Roman Bednarik, Niko Myller, Erkki Sutinen, and Markku Tukiainen. 2006. Analyzing individual differences in program comprehension. Technology Instruction Cognition and Learning 3, 3/4 (2006), 205.
- Roman Bednarik , Niko Myller , Erkki Sutinen , and Markku Tukiainen . 2006 . Program visualization: Comparing eye-tracking patterns with comprehension summaries and performance . In Proceedings of the 18th Annual Psychology of Programming Workshop. 66--82 . Roman Bednarik, Niko Myller, Erkki Sutinen, and Markku Tukiainen. 2006. Program visualization: Comparing eye-tracking patterns with comprehension summaries and performance. In Proceedings of the 18th Annual Psychology of Programming Workshop. 66--82.
- Roman Bednarik and Markku Tukiainen . 2004 . Visual attention and representation switching in Java program debugging: A study using eye movement tracking . In Proceedings of the 16th Annual Workshop of the Psychology of Programming Interest Group. 159--169 . Roman Bednarik and Markku Tukiainen. 2004. Visual attention and representation switching in Java program debugging: A study using eye movement tracking. In Proceedings of the 16th Annual Workshop of the Psychology of Programming Interest Group. 159--169.
- 10.1145/1028014.1028066 · DOI
- Roman Bednarik and Markku Tukiainen. 2005. Effects of display blurring on the behavior of novices and experts during program debugging. In CHI’05 Extended Abstracts on Human Factors in Computing Systems. ACM 1204--1207. Roman Bednarik and Markku Tukiainen. 2005. Effects of display blurring on the behavior of novices and experts during program debugging. In CHI’05 Extended Abstracts on Human Factors in Computing Systems. ACM 1204--1207. · doi:10.1145/1056808.1056877
- 10.1145/1117309.1117356 · DOI · Paper page
- Roman Bednarik and Markku Tukiainen . 2007 . Analysing and interpreting quantitative eye-tracking data in studies of programming: Phases of debugging with multiple representations . In Proceedings of the 19th Annual Workshop of the Psychology of Programming Interest Group (PPIG’07) . 158--172. Roman Bednarik and Markku Tukiainen. 2007. Analysing and interpreting quantitative eye-tracking data in studies of programming: Phases of debugging with multiple representations. In Proceedings of the 19th Annual Workshop of the Psychology of Programming Interest Group (PPIG’07). 158--172.
- 10.3758/BF03193158 · DOI
- 10.1145/1344471.1344497 · DOI · Paper page
- 10.1016/j.jvlc.2011.04.004 · DOI
- Alan F. Blackwell , Anthony R. Jansen , and Kim Marriott . 2000. Restricted focus viewer: A tool for tracking visual attention. Theory and Application of Diagrams . Springer , 162--177. Alan F. Blackwell, Anthony R. Jansen, and Kim Marriott. 2000. Restricted focus viewer: A tool for tracking visual attention. Theory and Application of Diagrams. Springer, 162--177.
- 10.1016/j.jss.2012.08.039 · DOI
- 10.5555/2820282.2820320 · doi:10.5555/2820282.2820320
- 10.1145/2578153.2578211 · DOI
- 10.1145/2094131.2094133 · DOI
- 10.1145/2632320.2632344 · DOI
- Teresa Busjahn , Carsten Schulte , Sascha Tamm , and Roman Bednarik (Eds.). 2015 . Eye Movements in Programming Education II: Analyzing the Novice’s Gaze. Technical Report. Freie Universität Berlin, Department of Mathematics and Computer Science , Berlin, Germany . 1--41 pages. Teresa Busjahn, Carsten Schulte, Sascha Tamm, and Roman Bednarik (Eds.). 2015. Eye Movements in Programming Education II: Analyzing the Novice’s Gaze. Technical Report. Freie Universität Berlin, Department of Mathematics and Computer Science, Berlin, Germany. 1--41 pages.
- Zoya Bylinskii , Michelle A. Borkin , Nam Wook Kim , Hanspeter Pfister , and Aude Oliva . 2015 . Eye fixation metrics for large scale evaluation and comparison of information visualizations . In Workshop on Eye Tracking and Visualization. Springer, 235--255 . Zoya Bylinskii, Michelle A. Borkin, Nam Wook Kim, Hanspeter Pfister, and Aude Oliva. 2015. Eye fixation metrics for large scale evaluation and comparison of information visualizations. In Workshop on Eye Tracking and Visualization. Springer, 235--255.
- 10.5555/77577.77580 · doi:10.5555/77577.77580
- 10.1145/3025453.3025573 · DOI
- 10.1016/j.scico.2012.01.004 · DOI
- 10.1076/csed.8.2.130.3820 · DOI
- Tore Dyba , Torgeir Dingsoyr , and Geir K. Hanssen . 2007. Applying systematic reviews to diverse study types: An experience report . In Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM’07) . IEEE, 225--234. Tore Dyba, Torgeir Dingsoyr, and Geir K. Hanssen. 2007. Applying systematic reviews to diverse study types: An experience report. In Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM’07). IEEE, 225--234.
- 10.1016/S0169-8141(98)00068-7 · DOI · Paper page
- 10.1145/1330598.1330691 · doi:10.1145/1330598.1330691
- Anabela Gomes and António José Mendes . 2007 . Learning to program—Difficulties and solutions . In Proceedings of the International Conference on Engineering Education (ICEE) , Vol. 2007 . Anabela Gomes and António José Mendes. 2007. Learning to program—Difficulties and solutions. In Proceedings of the International Conference on Engineering Education (ICEE), Vol. 2007.
- Yann-Gael Guéhéneuc , Huzefa Kagdi , and Jonathan I. Maletic . 2009. Working session: Using eye-tracking to understand program comprehension . In Proceedings of the IEEE 17th International Conference on Program Comprehension (ICPC’09) . IEEE, 278--279. Yann-Gael Guéhéneuc, Huzefa Kagdi, and Jonathan I. Maletic. 2009. Working session: Using eye-tracking to understand program comprehension. In Proceedings of the IEEE 17th International Conference on Program Comprehension (ICPC’09). IEEE, 278--279.
- 10.1109/eScience.2008.128 · DOI
- Ahmad Jbara and Dror G. Feitelson . 2015. How programmers read regular code: A controlled experiment using eye tracking . In Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension. IEEE, 244--254 . Ahmad Jbara and Dror G. Feitelson. 2015. How programmers read regular code: A controlled experiment using eye tracking. In Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension. IEEE, 244--254.
- Tomoko Kashima , Shimpei Matsumoto , and Shuichi Yamagishi . 2014. Proposal of a method to measure difficulty level of programming code with eye-tracking . In Human-Computer Interaction. Advanced Interaction Modalities and Techniques . Springer , 264--272. Tomoko Kashima, Shimpei Matsumoto, and Shuichi Yamagishi. 2014. Proposal of a method to measure difficulty level of programming code with eye-tracking. In Human-Computer Interaction. Advanced Interaction Modalities and Techniques. Springer, 264--272.
- 10.1109/TSE.2002.1027796 · DOI · Paper page
- 10.1109/HSI.2016.7529675 · doi:10.1109/hsi.2016.7529675
- 10.1145/2786805.2807561 · DOI
- 10.1016/j.edurev.2013.10.001 · DOI · Paper page
- 10.1023/A:1015362608943 · DOI · Paper page
- 10.1109/VISSOF.2009.5336413 · doi:10.1109/vissof.2009.5336413
- IT Chan Mow . 2008. Issues and difficulties in teaching novice computer programming . In Innovative Techniques in Instruction Technology, e-learning, e-assessment, and Education . Springer , 199--204. IT Chan Mow. 2008. Issues and difficulties in teaching novice computer programming. In Innovative Techniques in Instruction Technology, e-learning, e-assessment, and Education. Springer, 199--204.
- Seppo Nevalainen and Jorma Sajaniemi . 2004 . Comparison of three eye tracking devices in psychology of programming research . In Proceedings of the 16th Annual Psychology of Programming Interest Group Workshop. 151--158 . Seppo Nevalainen and Jorma Sajaniemi. 2004. Comparison of three eye tracking devices in psychology of programming research. In Proceedings of the 16th Annual Psychology of Programming Interest Group Workshop. 151--158.
- 10.1145/2857491.2857544 · DOI
- Patrick Peachock , Nicholas Iovino , and Bonita Sharif . 2017. Investigating Eye Movements in Natural Language and C++ Source Code—A Replication Experiment . Springer International Publishing , Cham , 206--218. DOI:http://dx.doi.org/10.1007/978-3-319-58628-1_17 10.1007/978-3-319-58628-1_17 Patrick Peachock, Nicholas Iovino, and Bonita Sharif. 2017. Investigating Eye Movements in Natural Language and C++ Source Code—A Replication Experiment. Springer International Publishing, Cham, 206--218. DOI:http://dx.doi.org/10.1007/978-3-319-58628-1_17 · DOI
- 10.5555/2227115.2227123 · doi:10.5555/2227115.2227123
- 10.1016/j.infsof.2015.03.007 · DOI · Paper page
- Razvan Petrusel and Jan Mendling . 2013. Eye-tracking the factors of process model comprehension tasks . In Advanced Information Systems Engineering . Springer , 224--239. Razvan Petrusel and Jan Mendling. 2013. Eye-tracking the factors of process model comprehension tasks. In Advanced Information Systems Engineering. Springer, 224--239.
- 10.1145/1344471.1344480 · doi:10.1145/1344471.1344480
- 10.1145/1833310.1833314 · doi:10.1145/1833310.1833314
- 10.1007/s10664-009-9125-9 · DOI
- 10.1109/ESEM.2015.7321188 · DOI
- Pablo Romero , Benedict Boulay , Richard Cox , and Rudi Lutz . 2003 . Java debugging strategies in multi-representational environments . In Proceedings of the 15th Annual Psychology of Programming Interest Group Workshop. 421--435 . Pablo Romero, Benedict Boulay, Richard Cox, and Rudi Lutz. 2003. Java debugging strategies in multi-representational environments. In Proceedings of the 15th Annual Psychology of Programming Interest Group Workshop. 421--435.
- 10.1109/SCAM.2015.7335412 · DOI
- 10.1109/ICPC.2011.40 · doi:10.1109/icpc.2011.40
- 10.1109/ICPC.2013.6613831 · DOI
- 10.1016/j.infsof.2015.06.008 · DOI · Paper page
- 10.1109/ICPC.2012.6240505 · DOI
- Bonita Sharif , Michael Falcone , and Jonathan I. Maletic . 2012. An eye-tracking study on the role of scan time in finding source code defects . In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA’12) . ACM, New York, 381--384. DOI:http://dx.doi.org/10.1145/2168556.2168642 10.1145/2168556.2168642 Bonita Sharif, Michael Falcone, and Jonathan I. Maletic. 2012. An eye-tracking study on the role of scan time in finding source code defects. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA’12). ACM, New York, 381--384. DOI:http://dx.doi.org/10.1145/2168556.2168642 · DOI · Paper page
- 10.1109/VISSOFT.2013.6650519 · DOI
- 10.1145/1987856.1987872 · DOI
- 10.1109/ICPC.2010.41 · DOI · Paper page
- 10.1109/ICSM.2010.5609582 · DOI
- Kshitij Sharma , Patrick Jermann , Marc-Antoine Nüssli , and Pierre Dillenbourg . 2013 . Understanding collaborative program comprehension: Interlacing gaze and dialogues . In Proceedings of Computer Supported Collaborative Learning (CSCL’13) . Kshitij Sharma, Patrick Jermann, Marc-Antoine Nüssli, and Pierre Dillenbourg. 2013. Understanding collaborative program comprehension: Interlacing gaze and dialogues. In Proceedings of Computer Supported Collaborative Learning (CSCL’13).
- 10.1145/1584322.1584334 · DOI · Paper page
- 10.1109/TSE.2005.97 · DOI · Paper page
- 10.1109/ICPC.2011.37 · DOI
- 10.1109/ICPC.2012.6240484 · DOI
- 10.1109/32.799942 · doi:10.1109/32.799942
- 10.1145/2578153.2578218 · DOI
- 10.1109/ICSEA.2008.49 · doi:10.1109/icsea.2008.49
- 10.1093/ietfec/e90-a.10.2290 · doi:10.1093/ietfec/e90-a.10.2290
- 10.1145/2993901.2993908 · DOI · Paper page
Cited by
Klaib, A. F., Alsrehin, N. O., Melhem, W. Y., Bashtawi, H. O., & Magableh, A. A. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114037 · Cited by 263
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196
“Eye tracking has been utilized in programming research for decades [7, 10, 11] and is mostly used to measure code comprehension and debugging [42], novice gaze patterns [8] and automatic skill level detection [1].”
Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641936 · Cited by 114
Mehdiyev, N., & Fettke, P. (2021) · Studies in Computational Intelligence · doi:10.1007/978-3-030-64949-4_1 · Cited by 78
An eye tracking study assessing source code readability rules for program comprehension
Park, K.-I., Johnson, J., Peterson, C. S., Yedla, N., Baysinger, I., Aponte, J., & Sharif, B. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10532-x · Cited by 9
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B., Kimmel, B., Wright, J., & Briggs, B. (2024) · arXiv · Cited by 196
“Eye tracking has been utilized in programming research for decades [7, 10, 11] and is mostly used to measure code comprehension and debugging [42], novice gaze patterns [8] and automatic skill level detection [1].”
(2025) · doi:10.1145/3723010.3723018 · Cited by 3
Self-citationFeeling Easier, Working Harder: Cognitive Load in AI-Assisted Code Comprehension
White, M., Rose, M., Buckman, M., McDonnell, A., Ortega, F., & Trinkenreich, B. (2026) · arXiv · doi:10.48550/arxiv.2611.12019
Metadata
- Authors
- Obaidellah, U., Al Haek, M., & Cheng, P. C.-H.
- Venue
- ACM Computing Surveys
- Published
- 2018-01-10
- Volume
- 51
- Issue
- 1
- Pages
- 1-58
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2783732587
- Cited by
- 162
- References
- 7
- FWCI
- 29.73
- Type
- journal article
- Language
- en
- Open access
- green
- Funders
- Ministry of Higher Education, Malaysia (FP062-2014A)
- Institut Pengurusan dan Pemantauan Penyelidikan, Universiti Malaya (RP030C-14AET)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.