Time-on-Task Metrics for Predicting Performance
Open accessAbstract
\emphTime-on-task is one key contributor to learning. However, how time-on-task is measured often varies, and is limited by the available data. In this work, we study two different time-on-task metrics---derived from programming process data---for predicting performance in an introductory programming course. The first metric, coarse-grained time-on-task, is based on students' submissions to programming assignments; the second, fine-grained time-on-task, is based on the keystrokes that students take while constructing their programs. Both types of time-on-task metrics have been used in prior work, and are supposedly designed to measure the same underlying feature: time-on-task. However, previous work has found that the correlation between these two metrics is not as high as one might expect. We build on that work by analyzing how well the two metrics work for predicting students' performance in an introductory programming course. Our results suggest that the correlation between the fine-grained time-on-task metric and both weekly exercise points and exam points is higher than the correlation between the coarse-grained time-on-task metric and weekly exercise points and exam points. Furthermore, we show that the fine-grained time-on-task metric is a better predictor of students' future success in the course exam than the coarse-grained time-on-task metric. We thus propose that future work utilizing time-on-task as a predictor of performance should use as fine-grained data as possible to measure time-on-task if such data is available.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Cites
Predicting academic performance: a systematic literature review
Hellas, A., Ihantola, P., Petersen, A., Ajanovski, V. V., Gutica, M., Hynninen, T., Knutas, A., Leinonen, J., Messom, C., & Liao, S. N. (2018) · Proceedings Companion of the 23rd Annual ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3293881.3295783 · Cited by 355 · see citing works
Self-citationMethods and tools for exploring novice compilation behaviour
Jadud, M. C. (2006) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/1151588.1151600 · Cited by 297 · see citing works
Exploring Machine Learning Methods to Automatically Identify Students in Need of Assistance
Ahadi, A., Lister, R., Haapala, H., & Vihavainen, A. (2015) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2787622.2787717 · Cited by 234 · see citing works
The Normalized Programming State Model
Carter, A. S., Hundhausen, C. D., & Adesope, O. (2015) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2787622.2787710 · Cited by 144 · see citing works
Scaffolding students' learning using test my code
Vihavainen, A., Vikberg, T., Luukkainen, M., & Pärtel, M. (2013) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2462476.2462501 · Cited by 135 · see citing works
Penetrating the black box of time-on-task estimation
Kovanović, V., Gašević, D., Dawson, S., Joksimovíc, S., Baker, R. S., & Hatala, M. (2015) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2723576.2723623 · Cited by 124 · see citing works
Comparing effective and ineffective behaviors of student programmers
Edwards, S. H., Snyder, J., Pérez-Quiñones, M. A., Allevato, A., Kim, D., & Tretola, B. (2009) · doi:10.1145/1584322.1584325 · Cited by 117 · see citing works
Does Time-on-task Estimation Matter? Implications on Validity of Learning Analytics Findings
Kovanović, V., Gašević, D., Dawson, S., Joksimovíc, S., & Baker, R. S. (2016) · Journal of Learning Analytics · doi:10.18608/jla.2015.23.6 · Cited by 117 · see citing works
Time on Task: A Naturalistic Study in Sixth-Grade Classrooms
Good, T. L., & Beckerman, T. M. (1978) · The Elementary School Journal · doi:10.1086/461101 · Cited by 116 · see citing works
Predicting student success using fine grain clicker data
Porter, L., Zingaro, D., & Lister, R. (2014) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2632320.2632354 · Cited by 110 · see citing works
Signal systems of lesson settings and the task-related behavior of preschool children.
Kounin, J. S., & Gump, P. V. (1974) · Journal of Educational Psychology · doi:10.1037/h0036748 · Cited by 108 · see citing works
Automatic Inference of Programming Performance and Experience from Typing Patterns
Leinonen, J., Longi, K., Klami, A., & Vihavainen, A. (2016) · Proceedings of the 47th ACM Technical Symposium on Computing Science Education · doi:10.1145/2839509.2844612 · Cited by 103 · see citing works
Self-citationA Robust Machine Learning Technique to Predict Low-performing Students
Liao, S. N., Zingaro, D., Thai, K., Alvarado, C., Griswold, W. G., & Porter, L. (2019) · ACM Transactions on Computing Education · doi:10.1145/3277569 · Cited by 91 · see citing works
Evaluating Neural Networks as a Method for Identifying Students in Need of Assistance
Castro-Wunsch, K., Ahadi, A., & Petersen, A. (2017) · doi:10.1145/3017680.3017792 · Cited by 73 · see citing works
Programming: predicting student success early in CS1. a re-validation and replication study
Quille, K., & Bergin, S. (2018) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3197091.3197101 · Cited by 63 · see citing works
Pereira, F. D., Fonseca, S. C., Oliveira, E. H. T., Cristea, A. I., Bellhauser, H., Rodrigues, L., Oliveira, D. B. F., Isotani, S., & Carvalho, L. S. G. (2021) · IEEE Access · doi:10.1109/access.2021.3105956 · Cited by 63 · see citing works
IDE-Based Learning Analytics for Computing Education
Hundhausen, C. D., Olivares, D. M., & Carter, A. S. (2017) · ACM Transactions on Computing Education · doi:10.1145/3105759 · Cited by 60 · see citing works
Predicting MOOCs Dropout Using Only Two Easily Obtainable Features from the First Week’s Activities
Alamri, A., Alshehri, M., Cristea, A. I., Pereira, F. D., Oliveira, E., Shi, L., & Stewart, C. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-22244-4_20 · Cited by 58 · see citing works
A fast measure for identifying at-risk students in computer science
Falkner, N., & Falkner, K. E. (2012) · doi:10.1145/2361276.2361288 · Cited by 51 · see citing works
Anderson, L. W., & Scott, C. C. (1978) · Journal of Teacher Education · doi:10.1177/002248717802900323 · Cited by 44 · see citing works
An exploration of error quotient in multiple contexts
Petersen, A., Spacco, J. W., & Vihavainen, A. (2015) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2828959.2828966 · Cited by 41 · see citing works
Comparison of Time Metrics in Programming
Leinonen, J., Leppänen, L., Ihantola, P., & Hellas, A. (2017) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3105726.3106181 · Cited by 41 · see citing works
Self-citationUsing Programming Process Data to Detect Differences in Students' Patterns of Programming
Carter, A. S., & Hundhausen, C. D. (2017) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3017680.3017785 · Cited by 31 · see citing works
A Study of Keystroke Data in Two Contexts
Edwards, J., Leinonen, J., & Hellas, A. (2020) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3328778.3366863 · Cited by 24 · see citing works
Self-citationLeinonen, J., Castro, F. E. V., & Hellas, A. (2021) · Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3430665.3456383 · Cited by 23 · see citing works
Self-citationNguyen, Q. (2020) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3375462.3375538 · Cited by 17 · see citing works
Understanding the Effects of Lecturer Intervention on Computer Science Student Behaviour
Szabo, C., Falkner, N. J. G., Knutas, A., & Dorodchi, M. (2018) · ITiCSE-WGR · doi:10.1145/3174781.3174787 · Cited by 7 · see citing works
Becker, B. A., Mooney, C., Kumar, A. N., & Russell, S. (2021) · Australasian Computing Education Conference · doi:10.1145/3441636.3442318 · Cited by 6 · see citing works
A Survey-based Exploration of Computer Science Student Perspectives on Mathematics
Sigurdson, N., & Petersen, A. (2019) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3287324.3287416 · Cited by 5 · see citing works
On the Differences in Time That Students Take to Write Solutions to Programming Problems
Fagerholm, F., & Hellas, A. (2020) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie44824.2020.9274237 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145 · see citing works
Self-citationInstructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study
Sheard, J., Denny, P., Hellas, A., Leinonen, J., Malmi, L., & Simon (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630880 · Cited by 54 · see citing works
Self-citationEvaluating Contextually Personalized Programming Exercises Created with Generative AI
Logacheva, E., Hellas, A., Prather, J. E., Sarsa, S., & Leinonen, J. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671103 · Cited by 63 · see citing works
Self-citationLehtinen, T., Haaranen, L., & Leinonen, J. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576129 · Cited by 17 · see citing works
Self-citationThe Impact of Solving Adaptive Parsons Problems with Common and Uncommon Solutions
Haynes-Magyar, C., & Ericson, B. J. (2022) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3564721.3564736 · Cited by 8 · see citing works
Leveraging response times in learning environments: opportunities and challenges
Pelánek, R. (2023) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-023-09386-7 · Cited by 7 · see citing works
Pullar-Strecker, Z., Pereira, F. D., Denny, P. C., Luxton-Reilly, A., & Leinonen, J. (2023) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545945.3569824 · Cited by 4 · see citing works
Self-citationContextualized Logging of On-Task and Off-Task Behaviours During Learning
Biedermann, D., Ciordas-Hertel, G.-P., Winter, M., Mordel, J., & Drachsler, H. (2023) · Journal of Learning Analytics · doi:10.18608/jla.2023.7837 · Cited by 3 · see citing works
Detecting Unsuccessful Students in Cybersecurity Exercises in Two Different Learning Environments
Švábenský, V., Tkáčik, K., Birdwell, A., Weiss, R., Baker, R. S., Čeleda, P., Vykopal, J., Mache, J., & Chattopadhyay, A. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893135 · Cited by 3 · see citing works
Seeing Program Output Improves Novice Learning Gains
Leinonen, J., Hellas, A., & Edwards, J. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588796 · Cited by 2 · see citing works
Self-citationAccurate Estimation of Time-on-Task While Programming
Hart, K., Warren, C. M., & Edwards, J. (2023) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545945.3569804 · Cited by 2 · see citing works
Assessing Computational Thinking Using Log Data: A Scoping Review
Mergianian, C., Hienonen, N., Dindar, M., & Vainikainen, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-03870-8_21 · Cited by 1 · see citing works
On the Opportunities of Large Language Models for Programming Process Data
Edwards, J., Hellas, A., & Leinonen, J. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716652 · Cited by 1 · see citing works
Self-citationExploring Error State in "Time-on-Task" Calculations at Scale
Chandarana, M., & Deitrick, E. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545947.3576282
Proposed Model for the Excellent Educational System: Overview of the Field
Bantilan, J. C. (2024) · Asian Journal of Education and Social Studies · doi:10.9734/ajess/2024/v50i61424
Time-on-task estimation for tasks lasting hours spread over multiple days
Hart, K., Warren, C. M., & Edwards, J. (2025) · PLoS ONE · doi:10.1371/journal.pone.0330758
Tateiwa, Y., Chen, G., Luo, Z., Uchiya, T., & Yin, C. (2026) · International Conference on Information Technology Based Higher Education and Training (ITHET) · doi:10.1109/ithet69978.2026.11585044
- Authors
- Leinonen, J., Castro, F. E. V., & Hellas, A.
- Affiliations
- Aalto University (FI)
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 53rd ACM Technical Symposium on Computer Science Education
- Publisher
- ACM
- Published
- 2022-02-22
- Pages
- 871-877
- OpenAlex
- W4293227716
- Cited by
- 18
- References
- 31
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 4 · 2023: 8 · 2022: 2
- FWCI
- 5.54
- Type
- conference paper
- Language
- en
- Open access
- green
- research.aalto.fi
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion)
- Aaltodoc (Aalto University) (submittedVersion, other-oa)
- Funders
- Media Industry Research Foundation of Finland
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- time on task, keystroke data, student performance prediction, introductory programming, programming exercises, exam performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
