Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Cited by 162

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

A Survey on the Usage of Eye-Tracking in Computer Programming

Open access
Obaidellah, Unaizah, Al Haek, Mohammed, Cheng, Peter C.-H. · 2019 · ACM Computing Surveys

Abstract

Traditional quantitative research methods of data collection in programming, such as questionnaires and interviews, are the most common approaches for researchers in this field. However, in recent years, eye-tracking has been on the rise as a new method of collecting evidence of visual attention and the cognitive process of programmers. Eye-tracking has been used by researchers in the field of programming to analyze and understand a variety of tasks such as comprehension and debugging. In this article, we will focus on reporting how experiments that used eye-trackers in programming research are conducted, and the information that can be collected from these experiments. In this mapping study, we identify and report on 63 studies, published between 1990 and June 2017, collected and gathered via manual search on digital libraries and databases related to computer science and computer engineering. Among the five main areas of research interest are program comprehension and debugging, which received an increased interest in recent years, non-code comprehension, collaborative programming, and requirements traceability research, which had the fewest number of publications due to possible limitations of the eye-tracking technology in this type of experiments. We find that most of the participants in these studies were students and faculty members from institutions of higher learning, and while they performed programming tasks on a range of programming languages and programming representations, we find Java language and Unified Modeling Language (UML) representation to be the most used materials. We also report on a range of eye-trackers and attention tracking tools that have been utilized, and find Tobii eye-trackers to be the most used devices by researchers.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3145904

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 7 references

Research works printed but not linked: 73

  1. 10.1007/s10664-014-9315-y · DOI
  2. 10.1080/10494820.2015.1073746 · DOI
  3. Roman Bednarik , Teresa Busjahn , and Carsten Schulte (Eds.). 2014 . Eye Movements in Programming Education: Analyzing the Experts Gaze. Technical Report . University of Eastern Finland , Joensuu, Finland . Roman Bednarik, Teresa Busjahn, and Carsten Schulte (Eds.). 2014. Eye Movements in Programming Education: Analyzing the Experts Gaze. Technical Report. University of Eastern Finland, Joensuu, Finland.
  4. 10.1109/VLHCC.2005.20 · doi:10.1109/vlhcc.2005.20
  5. Roman Bednarik , Niko Myller , Erkki Sutinen , and Markku Tukiainen . 2005 . Effects of experience on gaze behaviour during program animation . In Proceedings of the 17th Annual Psychology of Programming Interest Group Workshop. 49--61 . Roman Bednarik, Niko Myller, Erkki Sutinen, and Markku Tukiainen. 2005. Effects of experience on gaze behaviour during program animation. In Proceedings of the 17th Annual Psychology of Programming Interest Group Workshop. 49--61.
  6. Roman Bednarik , Niko Myller , Erkki Sutinen , and Markku Tukiainen . 2006 . Analyzing individual differences in program comprehension . Technology Instruction Cognition and Learning 3 , 3/4 (2006), 205 . Roman Bednarik, Niko Myller, Erkki Sutinen, and Markku Tukiainen. 2006. Analyzing individual differences in program comprehension. Technology Instruction Cognition and Learning 3, 3/4 (2006), 205.
  7. Roman Bednarik , Niko Myller , Erkki Sutinen , and Markku Tukiainen . 2006 . Program visualization: Comparing eye-tracking patterns with comprehension summaries and performance . In Proceedings of the 18th Annual Psychology of Programming Workshop. 66--82 . Roman Bednarik, Niko Myller, Erkki Sutinen, and Markku Tukiainen. 2006. Program visualization: Comparing eye-tracking patterns with comprehension summaries and performance. In Proceedings of the 18th Annual Psychology of Programming Workshop. 66--82.
  8. Roman Bednarik and Markku Tukiainen . 2004 . Visual attention and representation switching in Java program debugging: A study using eye movement tracking . In Proceedings of the 16th Annual Workshop of the Psychology of Programming Interest Group. 159--169 . Roman Bednarik and Markku Tukiainen. 2004. Visual attention and representation switching in Java program debugging: A study using eye movement tracking. In Proceedings of the 16th Annual Workshop of the Psychology of Programming Interest Group. 159--169.
  9. 10.1145/1028014.1028066 · DOI
  10. Roman Bednarik and Markku Tukiainen. 2005. Effects of display blurring on the behavior of novices and experts during program debugging. In CHI’05 Extended Abstracts on Human Factors in Computing Systems. ACM 1204--1207. Roman Bednarik and Markku Tukiainen. 2005. Effects of display blurring on the behavior of novices and experts during program debugging. In CHI’05 Extended Abstracts on Human Factors in Computing Systems. ACM 1204--1207. · doi:10.1145/1056808.1056877
  11. 10.1145/1117309.1117356 · DOI · Paper page
  12. Roman Bednarik and Markku Tukiainen . 2007 . Analysing and interpreting quantitative eye-tracking data in studies of programming: Phases of debugging with multiple representations . In Proceedings of the 19th Annual Workshop of the Psychology of Programming Interest Group (PPIG’07) . 158--172. Roman Bednarik and Markku Tukiainen. 2007. Analysing and interpreting quantitative eye-tracking data in studies of programming: Phases of debugging with multiple representations. In Proceedings of the 19th Annual Workshop of the Psychology of Programming Interest Group (PPIG’07). 158--172.
  13. 10.3758/BF03193158 · DOI
  14. 10.1145/1344471.1344497 · DOI
  15. 10.1016/j.jvlc.2011.04.004 · DOI
  16. Alan F. Blackwell , Anthony R. Jansen , and Kim Marriott . 2000. Restricted focus viewer: A tool for tracking visual attention. Theory and Application of Diagrams . Springer , 162--177. Alan F. Blackwell, Anthony R. Jansen, and Kim Marriott. 2000. Restricted focus viewer: A tool for tracking visual attention. Theory and Application of Diagrams. Springer, 162--177.
  17. 10.1016/j.jss.2012.08.039 · doi:10.1016/j.jss.2012.08.039
  18. 10.5555/2820282.2820320 · doi:10.5555/2820282.2820320
  19. 10.1145/2578153.2578211 · DOI
  20. 10.1145/2094131.2094133 · DOI
  21. 10.1145/2632320.2632344 · DOI
  22. Teresa Busjahn , Carsten Schulte , Sascha Tamm , and Roman Bednarik (Eds.). 2015 . Eye Movements in Programming Education II: Analyzing the Novice’s Gaze. Technical Report. Freie Universität Berlin, Department of Mathematics and Computer Science , Berlin, Germany . 1--41 pages. Teresa Busjahn, Carsten Schulte, Sascha Tamm, and Roman Bednarik (Eds.). 2015. Eye Movements in Programming Education II: Analyzing the Novice’s Gaze. Technical Report. Freie Universität Berlin, Department of Mathematics and Computer Science, Berlin, Germany. 1--41 pages.
  23. Zoya Bylinskii , Michelle A. Borkin , Nam Wook Kim , Hanspeter Pfister , and Aude Oliva . 2015 . Eye fixation metrics for large scale evaluation and comparison of information visualizations . In Workshop on Eye Tracking and Visualization. Springer, 235--255 . Zoya Bylinskii, Michelle A. Borkin, Nam Wook Kim, Hanspeter Pfister, and Aude Oliva. 2015. Eye fixation metrics for large scale evaluation and comparison of information visualizations. In Workshop on Eye Tracking and Visualization. Springer, 235--255.
  24. 10.5555/77577.77580 · doi:10.5555/77577.77580
  25. 10.1145/3025453.3025573 · DOI
  26. 10.1016/j.scico.2012.01.004 · DOI
  27. 10.1076/csed.8.2.130.3820 · DOI
  28. Tore Dyba , Torgeir Dingsoyr , and Geir K. Hanssen . 2007. Applying systematic reviews to diverse study types: An experience report . In Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM’07) . IEEE, 225--234. Tore Dyba, Torgeir Dingsoyr, and Geir K. Hanssen. 2007. Applying systematic reviews to diverse study types: An experience report. In Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM’07). IEEE, 225--234.
  29. 10.1016/S0169-8141(98)00068-7 · DOI · Paper page
  30. 10.1145/1330598.1330691 · doi:10.1145/1330598.1330691
  31. Anabela Gomes and António José Mendes . 2007 . Learning to program—Difficulties and solutions . In Proceedings of the International Conference on Engineering Education (ICEE) , Vol. 2007 . Anabela Gomes and António José Mendes. 2007. Learning to program—Difficulties and solutions. In Proceedings of the International Conference on Engineering Education (ICEE), Vol. 2007.
  32. Yann-Gael Guéhéneuc , Huzefa Kagdi , and Jonathan I. Maletic . 2009. Working session: Using eye-tracking to understand program comprehension . In Proceedings of the IEEE 17th International Conference on Program Comprehension (ICPC’09) . IEEE, 278--279. Yann-Gael Guéhéneuc, Huzefa Kagdi, and Jonathan I. Maletic. 2009. Working session: Using eye-tracking to understand program comprehension. In Proceedings of the IEEE 17th International Conference on Program Comprehension (ICPC’09). IEEE, 278--279.
  33. 10.1109/eScience.2008.128 · DOI
  34. Ahmad Jbara and Dror G. Feitelson . 2015. How programmers read regular code: A controlled experiment using eye tracking . In Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension. IEEE, 244--254 . Ahmad Jbara and Dror G. Feitelson. 2015. How programmers read regular code: A controlled experiment using eye tracking. In Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension. IEEE, 244--254.
  35. Tomoko Kashima , Shimpei Matsumoto , and Shuichi Yamagishi . 2014. Proposal of a method to measure difficulty level of programming code with eye-tracking . In Human-Computer Interaction. Advanced Interaction Modalities and Techniques . Springer , 264--272. Tomoko Kashima, Shimpei Matsumoto, and Shuichi Yamagishi. 2014. Proposal of a method to measure difficulty level of programming code with eye-tracking. In Human-Computer Interaction. Advanced Interaction Modalities and Techniques. Springer, 264--272.
  36. 10.1109/TSE.2002.1027796 · DOI · Paper page
  37. 10.1109/HSI.2016.7529675 · doi:10.1109/hsi.2016.7529675
  38. 10.1145/2786805.2807561 · DOI
  39. 10.1016/j.edurev.2013.10.001 · DOI · Paper page
  40. 10.1023/A:1015362608943 · DOI · Paper page
  41. 10.1109/VISSOF.2009.5336413 · doi:10.1109/vissof.2009.5336413
  42. IT Chan Mow . 2008. Issues and difficulties in teaching novice computer programming . In Innovative Techniques in Instruction Technology, e-learning, e-assessment, and Education . Springer , 199--204. IT Chan Mow. 2008. Issues and difficulties in teaching novice computer programming. In Innovative Techniques in Instruction Technology, e-learning, e-assessment, and Education. Springer, 199--204.
  43. Seppo Nevalainen and Jorma Sajaniemi . 2004 . Comparison of three eye tracking devices in psychology of programming research . In Proceedings of the 16th Annual Psychology of Programming Interest Group Workshop. 151--158 . Seppo Nevalainen and Jorma Sajaniemi. 2004. Comparison of three eye tracking devices in psychology of programming research. In Proceedings of the 16th Annual Psychology of Programming Interest Group Workshop. 151--158.
  44. 10.1145/2857491.2857544 · DOI
  45. Patrick Peachock , Nicholas Iovino , and Bonita Sharif . 2017. Investigating Eye Movements in Natural Language and C++ Source Code—A Replication Experiment . Springer International Publishing , Cham , 206--218. DOI:http://dx.doi.org/10.1007/978-3-319-58628-1_17 10.1007/978-3-319-58628-1_17 Patrick Peachock, Nicholas Iovino, and Bonita Sharif. 2017. Investigating Eye Movements in Natural Language and C++ Source Code—A Replication Experiment. Springer International Publishing, Cham, 206--218. DOI:http://dx.doi.org/10.1007/978-3-319-58628-1_17 · DOI
  46. 10.5555/2227115.2227123 · doi:10.5555/2227115.2227123
  47. 10.1016/j.infsof.2015.03.007 · DOI · Paper page
  48. Razvan Petrusel and Jan Mendling . 2013. Eye-tracking the factors of process model comprehension tasks . In Advanced Information Systems Engineering . Springer , 224--239. Razvan Petrusel and Jan Mendling. 2013. Eye-tracking the factors of process model comprehension tasks. In Advanced Information Systems Engineering. Springer, 224--239.
  49. 10.1145/1344471.1344480 · doi:10.1145/1344471.1344480
  50. 10.1145/1833310.1833314 · doi:10.1145/1833310.1833314
  51. 10.1007/s10664-009-9125-9 · DOI
  52. 10.1109/ESEM.2015.7321188 · DOI
  53. Pablo Romero , Benedict Boulay , Richard Cox , and Rudi Lutz . 2003 . Java debugging strategies in multi-representational environments . In Proceedings of the 15th Annual Psychology of Programming Interest Group Workshop. 421--435 . Pablo Romero, Benedict Boulay, Richard Cox, and Rudi Lutz. 2003. Java debugging strategies in multi-representational environments. In Proceedings of the 15th Annual Psychology of Programming Interest Group Workshop. 421--435.
  54. 10.1109/SCAM.2015.7335412 · DOI
  55. 10.1109/ICPC.2011.40 · doi:10.1109/icpc.2011.40
  56. 10.1109/ICPC.2013.6613831 · DOI
  57. 10.1016/j.infsof.2015.06.008 · DOI · Paper page
  58. 10.1109/ICPC.2012.6240505 · DOI
  59. Bonita Sharif , Michael Falcone , and Jonathan I. Maletic . 2012. An eye-tracking study on the role of scan time in finding source code defects . In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA’12) . ACM, New York, 381--384. DOI:http://dx.doi.org/10.1145/2168556.2168642 10.1145/2168556.2168642 Bonita Sharif, Michael Falcone, and Jonathan I. Maletic. 2012. An eye-tracking study on the role of scan time in finding source code defects. In Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA’12). ACM, New York, 381--384. DOI:http://dx.doi.org/10.1145/2168556.2168642 · DOI
  60. 10.1109/VISSOFT.2013.6650519 · DOI
  61. 10.1145/1987856.1987872 · DOI
  62. 10.1109/ICPC.2010.41 · DOI
  63. 10.1109/ICSM.2010.5609582 · DOI
  64. Kshitij Sharma , Patrick Jermann , Marc-Antoine Nüssli , and Pierre Dillenbourg . 2013 . Understanding collaborative program comprehension: Interlacing gaze and dialogues . In Proceedings of Computer Supported Collaborative Learning (CSCL’13) . Kshitij Sharma, Patrick Jermann, Marc-Antoine Nüssli, and Pierre Dillenbourg. 2013. Understanding collaborative program comprehension: Interlacing gaze and dialogues. In Proceedings of Computer Supported Collaborative Learning (CSCL’13).
  65. 10.1145/1584322.1584334 · DOI · Paper page
  66. 10.1109/TSE.2005.97 · DOI · Paper page
  67. 10.1109/ICPC.2011.37 · DOI
  68. 10.1109/ICPC.2012.6240484 · DOI
  69. 10.1109/32.799942 · doi:10.1109/32.799942
  70. 10.1145/2578153.2578218 · DOI
  71. 10.1109/ICSEA.2008.49 · doi:10.1109/icsea.2008.49
  72. 10.1093/ietfec/e90-a.10.2290 · doi:10.1093/ietfec/e90-a.10.2290
  73. 10.1145/2993901.2993908 · DOI · Paper page

Cited by

See all 162

Metadata

Authors
Obaidellah, U., Al Haek, M., & Cheng, P. C.-H.
Venue
ACM Computing Surveys
Published
2018-01-10
Volume
51
Issue
1
Pages
1-58
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
OpenAlex
W2783732587
Cited by
162
References
7
FWCI
29.73
Type
journal article
Language
en
Open access
green
Funders
  • Ministry of Higher Education, Malaysia (FP062-2014A)
  • Institut Pengurusan dan Pemantauan Penyelidikan, Universiti Malaya (RP030C-14AET)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Survey on the Usage of Eye-Tracking in Computer Programming. https://app.cochat.ai/papers/10.1145/3145904
The paper
Obaidellah, U., Al Haek, M., & Cheng, P. C.-H. (2019). A Survey on the Usage of Eye-Tracking in Computer Programming. ACM Computing Surveys. https://doi.org/10.1145/3145904

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.