Automatic Inference of Programming Performance and Experience from Typing Patterns
Open accessAbstract
Studies on retention and success in introductory programming course have suggested that previous programming experience contributes to students' course outcomes. If such background information could be automatically distilled from students' working process, additional guidance and support mechanisms could be provided even to those, who do not wish to disclose such information. In this study, we explore methods for automatically distinguishing novice programmers from more experienced programmers using fine-grained source code snapshot data. We approach the issue by partially replicating a previous study that used students' keystroke latencies as a proxy to introductory programming course outcomes, and follow this by an exploration of machine learning methods to separate those students with little to no previous programming experience from those with more experience. Our results confirm that students' keystroke latencies can be used as a metric for measuring course outcomes. At the same time, our results show that students programming experience can be identified to some extent from keystroke latency data, which means that such data has potential as a source of information for customizing the students' learning experience.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some work has used fine-grained programming process logs from instrumented IDEs to uncover student work patterns and to visualize student programming processes.
“Some work has looked into using fine-grained programming process logs gathered from instrumented IDEs to uncover student work patterns [21-23, 26, 40] and to visualize student programming processes [10,14,35,38].”
From Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study (2024) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · see citing works
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
Powers, D. M. W. (2020) · arXiv · doi:10.48550/arxiv.2010.16061 · Cited by 5,440 · see citing works
Educational Data Mining: A review of the state of the art
Romero, C., & Ventura, S. (2010) · IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews) · doi:10.1109/tsmcc.2010.2053532 · Cited by 2,251 · see citing works
Contributing to success in an introductory computer science course
Wilson, B. C., & Shrock, S. A. (2001) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/364447.364581 · Cited by 376 · see citing works
Identifying emotional states using keystroke dynamics
Epp, C., Lippold, M., & Mandryk, R. L. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979046 · Cited by 364 · see citing works
Authentication by Keystroke Timing: Some Preliminary Results
Gaines, R. S., Lisowski, W., Press, S. J., & Shapiro, N. Z. (1980) · Defense Technical Information Center (DTIC) · Cited by 347 · see citing works
Methods and tools for exploring novice compilation behaviour
Jadud, M. C. (2006) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/1151588.1151600 · Cited by 297 · see citing works
Typing patterns: a key to user identification
Peacock, A., Ke, X., & Wilkerson, M. D. (2004) · IEEE Security & Privacy · doi:10.1109/msp.2004.89 · Cited by 258 · see citing works
Biometric personal authentication using keystroke dynamics: A review
Karnan, M., Akila, M., & Krishnaraj, N. (2010) · Applied Soft Computing · doi:10.1016/j.asoc.2010.08.003 · Cited by 242 · see citing works
Bergin, S., & Reilly, R. B. (2005) · ACM SIGCSE Bulletin · doi:10.1145/1047124.1047480 · Cited by 242 · see citing works
Exploring Machine Learning Methods to Automatically Identify Students in Need of Assistance
Ahadi, A., Lister, R., Haapala, H., & Vihavainen, A. (2015) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2787622.2787717 · Cited by 234 · see citing works
Self-citationAutomated stress detection using keystroke and linguistic features: An exploratory study
Vizer, L. M., Zhou, L., & Sears, A. (2009) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2009.07.005 · Cited by 224 · see citing works
Contributing to success in an introductory computer science course
Wilson, B. C., & Shrock, S. A. (2001) · ACM SIGCSE Bulletin · doi:10.1145/366413.364581 · Cited by 207 · see citing works
Watson, C., Li, F. W. B., & Godwin, J. L. (2013) · IEEE International Conference on Advanced Learning Technologies (ICALT) · doi:10.1109/icalt.2013.99 · Cited by 206 · see citing works
What best predicts computer proficiency?
Evans, G. E., & Simkin, M. (1989) · Communications of the ACM · doi:10.1145/68814.68817 · Cited by 204 · see citing works
Does it help to have some programming experience before beginning a computing degree program?
Hagan, D., & Markham, S. (2000) · doi:10.1145/343048.343063 · Cited by 179 · see citing works
Watson, C., Li, F. W. B., & Godwin, J. L. (2014) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2538862.2538930 · Cited by 176 · see citing works
The Normalized Programming State Model
Carter, A. S., Hundhausen, C. D., & Adesope, O. (2015) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2787622.2787710 · Cited by 144 · see citing works
Scaffolding students' learning using test my code
Vihavainen, A., Vikberg, T., Luukkainen, M., & Pärtel, M. (2013) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2462476.2462501 · Cited by 135 · see citing works
Self-citationEducational Psychology a Contemporary Approach
Borich, G. D. (1995) · Medical Entomology and Zoology · Cited by 112 · see citing works
Bixler, R., & D'Mello, S. K. (2013) · International Conference on Intelligent User Interfaces · doi:10.1145/2449396.2449426 · Cited by 106 · see citing works
Factors Affecting Course Outcomes in Introductory Programming
Wiedenbeck, S., LaBelle, D., State, P., University-Delaware County, & Kain, V. N. R. (2004) · Cited by 101 · see citing works
Villani, M., Tappert, C. C., Ngo, G., Simone, J., Fort, H. S., & Cha, S.-H. (2006) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2006.115 · Cited by 96 · see citing works
Does it help to have some programming experience before beginning a computing degree program?
Hagan, D., & Markham, S. (2000) · ACM SIGCSE Bulletin · doi:10.1145/353519.343063 · Cited by 83 · see citing works
A Long-Term Trial of Keystroke Profiling Using Digraph, Trigraph and Keyword Latencies
Haskell‐Dowland, P., & Furnell, S. (2004) · Springer eBooks · doi:10.1007/1-4020-8143-x_18 · Cited by 70 · see citing works
A Keystroke Dynamics Based System for User Identification
Rybnik, M., Tabędzki, M., & Saeed, K. (2008) · doi:10.1109/cisim.2008.8 · Cited by 63 · see citing works
Free vs. transcribed text for keystroke-dynamics evaluations
Killourhy, K. S., & Maxion, R. A. (2012) · doi:10.1145/2379616.2379617 · Cited by 50 · see citing works
Geek genes, prior knowledge, stumbling points and learning edge momentum
Ahadi, A., & Lister, R. (2013) · doi:10.1145/2493394.2493416 · Cited by 45 · see citing works
Identification of programmers from typing patterns
Longi, K., Leinonen, J., Nygren, H., Salmi, J., Arto Klami, A., & Vihavainen, A. (2015) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2828959.2828960 · Cited by 44 · see citing works
Self-citationAn investigation into keystroke latency metrics as an indicator of programming performance
Thomas, R. C., Karahasanović, A., & Kennedy, G. E. (2005) · Australasian Computing Education Conference · Cited by 35 · see citing works
Management, structures and tools to scale up personal advising in large programming courses
Kurhila, J., & Vihavainen, A. (2011) · Conference on Information Technology Education · doi:10.1145/2047594.2047596 · Cited by 35 · see citing works
Self-citationEducational psychology: A contemporary approach, 2nd ed.
Borich, G. D., & Tombari, M. L. (1997) · Cited by 23 · see citing works
Psychological Tests and Selection of Computer Programmers
Rowan, T. C. (1957) · Journal of the ACM · doi:10.1145/320881.320891 · Cited by 22 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- S. Bergin and R. Reilly. Programming: factors that influence success. ACM SIGCSE Bull., 37(1):411-415, 2005.
Cited by
Sorted by citations
Predicting academic performance: a systematic literature review
Hellas, A., Ihantola, P., Petersen, A., Ajanovski, V. V., Gutica, M., Hynninen, T., Knutas, A., Leinonen, J., Messom, C., & Liao, S. N. (2018) · Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3293881.3295783 · Cited by 355 · see citing works
Self-citationPereira, F. D., Fonseca, S. C., Oliveira, E. H. T., Cristea, A. I., Bellhauser, H., Rodrigues, L., Oliveira, D. B. F., Isotani, S., & Carvalho, L. S. G. (2021) · IEEE Access · doi:10.1109/access.2021.3105956 · Cited by 63 · see citing works
Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study
Sheard, J., Denny, P., Hellas, A., Leinonen, J., Malmi, L., & Simon (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630880 · Cited by 54 · see citing works
Self-citationReplication in computing education research: researcher attitudes and experiences
Ahadi, A., Hellas, A., Ihantola, P., Korhonen, A., & Petersen, A. (2016) · Proceedings of the 16th Koli Calling International Conference on Computing Education Research · doi:10.1145/2999541.2999554 · Cited by 48 · see citing works
Artificial intelligence in computer programming education: A systematic literature review
Manorat, P., Tuarob, S., & Pongpaichet, S. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2025.100403 · Cited by 27 · see citing works
Time-on-Task Metrics for Predicting Performance
Leinonen, J., Castro, F. E. V., & Hellas, A. (2022) · Proceedings of the 53rd ACM Technical Symposium on Computer Science Education · doi:10.1145/3478431.3499359 · Cited by 18 · see citing works
Self-citationLehtinen, T., Haaranen, L., & Leinonen, J. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576129 · Cited by 17 · see citing works
Self-citationMorning or Evening? An Examination of Circadian Rhythms of CS1 Students
Zavgorodniaia, A., Shrestha, R., Leinonen, J., Hellas, A., & Edwards, J. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1109/icse-seet52601.2021.00036 · Cited by 13 · see citing works
Self-citationComputing Education Research Today
Fincher, S. A., Tenenberg, J., Dorn, B., Hundhausen, C., McCartney, R., & Murphy, L. (2019) · The Cambridge Handbook of Computing Education Research · doi:10.1017/9781108654555.003 · Cited by 7 · see citing works
- Authors
- Leinonen, J., Longi, K., Klami, A., & Vihavainen, A.
- Affiliations
- University of Helsinki (FI)
- University of Helsinki, Helsinki, Finland
- ORCID
- Leinonen, Juho
- Venue
- Proceedings of the 47th ACM Technical Symposium on Computing Science Education
- Publisher
- ACM
- Published
- 2016-02-17
- Pages
- 132-137
- OpenAlex
- W2275142089
- Cited by
- 103
- References
- 33
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 4 · 2023: 10 · 2022: 15 · 2021: 4 · 2020: 10 · 2019: 39 · 2018: 5 · 2017: 10 · 2016: 2
- FWCI
- 14.89
- Type
- conference paper
- Language
- en
- Open access
- green
- hdl.handle.net
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion)
- Työväentutkimus Vuosikirja (submittedVersion, other-oa)
- Funders
- Academy of Finland (Project 1266969 and COIN Centre of Excellence)
- Finnish Funding Agency for Innovation (Re:Know)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- keystroke latency, typing patterns, personalized learning support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
