On the Suitability of Real-Time Assessment of Programming Proficiency using Gaze Properties
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Evaluating programming proficiency has become more relevant as the demand for coding skills has increased. Current methods, such as questionnaires or interviews, are methods that lack intuition, flexibility, and real-time capabilities. In our work, we investigate eye gaze behavior as an estimate for skill assessment. Specifically, we conducted a study (N=14) using an eye tracker to analyze the participants' abilities to understand source code by presenting them with a series of programs. We evaluated their eye movements based on common eye tracking metrics and identified mutual task-solving strategies among the participants. While we cannot relate these indicators to programming proficiency directly, this study serves as an evaluation of real-time methods for evaluating programming proficiency.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Previous work has also shown that smooth pursuit eye movements deviate from a given trajectory in situations with a higher cognitive demand [133] while being less prone to changes in light intensity (cf.pupil diameter).”
From A Survey on Measuring Cognitive Workload in Human-Computer Interaction (2023) · cites this paper as [133]
In the citing paper
“However, previous research has shown that various eye movements can also indicate increased cognitive demand [63,131,133] or derive the user proficiency during interaction [151].”
From A Survey on Measuring Cognitive Workload in Human-Computer Interaction (2023) · cites this paper as [133]
Citations
Cites
Eye movements in reading and information processing: 20 years of research
Rayner, K. (1998) · Psychological Bulletin · doi:10.1037/0033-2909.124.3.372 · Cited by 7,979
Eye Tracking in Human-Computer Interaction and Usability Research
Jacob, R. J. K., & Karn, K. S. (2003) · The Mind's Eye · doi:10.1016/b978-044451020-4/50031-1 · Cited by 1,567
An eye-tracking methodology for characterizing program comprehension processes
Bednarik, R., & Tukiainen, M. (2006) · Proceedings of the 2006 symposium on Eye tracking research & applications - ETRA '06 · doi:10.1145/1117309.1117356 · Cited by 197
Predicting performance in an introductory computer science course
Butcher, D. F., & Muth, W. A. (1985) · Communications of the ACM · doi:10.1145/3166.3167 · Cited by 120
Measuring programming experience
Feigenspan, J., Kästner, C., Liebig, J., Apel, S., & Hanenberg, S. (2012) · 2012 20th IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc.2012.6240511 · Cited by 106
Your Eyes Tell: Leveraging Smooth Pursuit for Assessing Cognitive Workload
Kosch, T., Hassib, M., Woźniak, P. W., Buschek, D., & Alt, F. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174010 · Cited by 79
Self-citationRecognition of understanding level and language skill using measurements of reading behavior
Martínez-Gómez, P., & Aizawa, A. (2014) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/2557500.2557546 · Cited by 68
Robust Gaze Features for Enabling Language Proficiency Awareness
Karolus, J., Woźniak, P. W., Chuang, L. L., & Schmidt, A. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025601 · Cited by 21
Self-citation
Cited by
A Survey on Measuring Cognitive Workload in Human-Computer Interaction
Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., & Woźniak, P. W. (2023) · ACM Computing Surveys · doi:10.1145/3582272 · Cited by 304
“However, previous research has shown that various eye movements can also indicate increased cognitive demand [63, 131, 133] or derive the user proficiency during interaction [151].”
Self-citation
Metadata
- Authors
- Liang, C., Karolus, J., Kosch, T., & Schmidt, A.
- Affiliations
- Tufts University (US)
- Ludwig-Maximilians-Universität München (DE)
- Venue
- Proceedings of the 7th ACM International Symposium on Pervasive Displays
- Publisher
- ACM
- Published
- 2018-06-06
- Pages
- 1-2
- OpenAlex
- W2804047300
- Cited by
- 3
- References
- 8
- Citations by year
- 2024: 1 · 2023: 1 · 2020: 1
- FWCI
- 0.20
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM International Symposium on Pervasive Displays (PerDis) (publishedVersion)
- Funders
- NSF (IRES)
- BMBF (16SV7599K)
- SFB (TRR 161)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- eye tracking, gaze behavior, source code comprehension, eye movement metrics, programming education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: On the Suitability of Real-Time Assessment of Programming Proficiency using Gaze Properties. https://app.cochat.ai/papers/10.1145/3205873.3210702
- The paper
- Liang, C., Karolus, J., Kosch, T., & Schmidt, A. (2018). On the Suitability of Real-Time Assessment of Programming Proficiency using Gaze Properties. Proceedings of the 7th ACM International Symposium on Pervasive Displays. https://doi.org/10.1145/3205873.3210702
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.