How Widely Can Prediction Models Be Generalized? Performance Prediction in Blended Courses
Open accessAbstract
Blended courses that mix in-person instruction with online platforms are increasingly common in secondary education. These platforms record a rich amount of data on students' study habits and social interactions. Prior research has shown that these metrics are correlated with students performance in face-to-face classes. However, predictive models for blended courses are still limited and have not yet succeeded at early prediction or cross-class predictions, even for repeated offerings of the same course. In this paper, we use data from two offerings of two different undergraduate courses to train and evaluate predictive models of student performance based on persistent student characteristics including study habits and social interactions. We analyze the performance of these models on the same offering, on different offerings of the same course, and across courses to see how well they generalize. We also evaluate the models on different segments of the courses to determine how early reliable predictions can be made. This paper tells us in part how much data is required to make robust predictions and how cross-class data may be used, or not, to boost model performance. The results of this study will help us better understand how similar the study habits, social activities, and the teamwork styles are across semesters for students in each performance category. These trained models also provide an avenue to improve our existing support platforms to better support struggling students early in the semester with the goal of providing timely intervention.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Generalizing performance prediction to non-traditional modes of learning such as hybrid or blended learning and online learning has benefited from the inclusion of additional information sources such as web-browsing information, discussion forum activity and student study habits.
“The process of generalizing performance prediction to non-traditional modes of learning such as hybrid or blended learning and on-line learning has benefitted from the inclusion of additional information sources such as web-browsing information (Trakunphutthirak et al., 2019), discussion forum activity and student study habits (Gitinabard et al., 2019).”
From Proactive and reactive engagement of artificial intelligence methods for education: a review (2023) · cites this paper as Gitinabard et al., 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Lave, J., & Wenger, E. (1991) · Cambridge University Press eBooks · doi:10.1017/cbo9780511815355 · Cited by 33,219 · see citing works
Communities of Practice: Learning, Meaning, and Identity
Wenger, E. (1998) · doi:10.1017/cbo9780511803932 · Cited by 23,500 · see citing works
Centrality in social networks conceptual clarification
Freeman, L. C. (1978) · Social Networks · doi:10.1016/0378-8733(78)90021-7 · Cited by 17,062 · see citing works
Authoritative sources in a hyperlinked environment
Kleinberg, J. M. (1999) · Journal of the ACM · doi:10.1145/324133.324140 · Cited by 9,032 · see citing works
The use of ranks to avoid the assumption of normality implicit in the analysis of variance
Friedman, M. (1937) · Journal of the American Statistical Association · doi:10.1080/01621459.1937.10503522 · Cited by 7,005 · see citing works
Mining LMS data to develop an “early warning system” for educators: A proof of concept
Macfadyen, L. P., & Dawson, S. (2010) · Computers & Education · doi:10.1016/j.compedu.2009.09.008 · Cited by 1,045 · see citing works
A survey of current research on online communities of practice
Johnson, C. M. (2001) · The Internet and Higher Education · doi:10.1016/s1096-7516(01)00047-1 · Cited by 699 · see citing works
Rank Methods for Combination of Independent Experiments in Analysis of Variance
Hodges, J. L., & Lehmann, E. L. (1962) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177704575 · Cited by 398 · see citing works
“Predicting MOOC dropout over weeks using machine learning methods,”
Kloft, M., Stiehler, F., Zheng, Z., & Pinkwart, N. (2014) · Proceedings of the EMNLP 2014 Workshop on Analysis of Large Scale Social Interaction in MOOCs · doi:10.3115/v1/w14-4111 · Cited by 325 · see citing works
Using learning analytics to assess students' behavior in open-ended programming tasks
Blikstein, P. (2011) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2090116.2090132 · Cited by 294 · see citing works
Temporal Models for Predicting Student Dropout in Massive Open Online Courses
Mi, F., & Yeung, D. (2015) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2015.174 · Cited by 287 · see citing works
Social presence in online discussions as a process predictor of academic performance
Joksimovíc, S., Gašević, D., Kovanović, V., Riecke, B. E., & Hatala, M. (2015) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12107 · Cited by 284 · see citing works
STAC: A web platform for the comparison of algorithms using statistical tests
Rodriguez-Fdez, I., Canosa, A., Mucientes, M., & Bugarín, A. (2015) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzz-ieee.2015.7337889 · Cited by 230 · see citing works
Understanding MOOC students: motivations and behaviours indicative of MOOC completion
Pursel, B. K., Zhang, L., Jablokow, K., Choi, G. W., & Velegol, D. (2016) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12131 · Cited by 222 · see citing works
Chao, P.-Y. (2016) · Computers & Education · doi:10.1016/j.compedu.2016.01.010 · Cited by 211 · see citing works
Watson, C., Li, F. W. B., & Godwin, J. L. (2014) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2538862.2538930 · Cited by 176 · see citing works
MOOC performance prediction via clickstream data and social learning networks
Brinton, C. G., & Chiang, M. (2015) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom.2015.7218617 · Cited by 144 · see citing works
The Normalized Programming State Model
Carter, A. S., Hundhausen, C. D., & Adesope, O. (2015) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/2787622.2787710 · Cited by 144 · see citing works
Social factors that contribute to attrition in MOOCs
Rosé, C. P., Carlson, R., Yang, D., Wen, M., Resnick, L. Β., Goldman, P., & Sherer, J. Z. (2014) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2556325.2567879 · Cited by 140 · see citing works
Exploring the Effect of Confusion in Discussion Forums of Massive Open Online Courses
Yang, D., Wen, M.-M., Howley, I., Kraut, R. E., & Rosé, C. (2015) · ACM Conference on Learning @ Scale · doi:10.1145/2724660.2724677 · Cited by 137 · see citing works
Penetrating the black box of time-on-task estimation
Kovanović, V., Gašević, D., Dawson, S., Joksimovíc, S., Baker, R. S., & Hatala, M. (2015) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2723576.2723623 · Cited by 124 · see citing works
Transfer Learning for Predictive Models in Massive Open Online Courses
Boyer, S., & Veeramachaneni, K. K. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-19773-9_6 · Cited by 102 · see citing works
Borthick, A. F., & Jones, D. R. (2000) · Issues in Accounting Education · doi:10.2308/iace.2000.15.2.181 · Cited by 96 · see citing works
Tracking Students’ Cognitive Processes During Program Debugging—An Eye-Movement Approach
Lin, Y.-T., Wu, C.-C., Hou, T.-Y., Lin, Y.-C., Yang, F., & Chang, C.-H. (2015) · IEEE Transactions on Education · doi:10.1109/te.2015.2487341 · Cited by 85 · see citing works
An analysis of patterns of debugging among novice computer science students
Ahmadzadeh, M., Elliman, D., & Higgins, C. (2005) · ACM SIGCSE Bulletin · doi:10.1145/1151954.1067472 · Cited by 85 · see citing works
A social network analysis of student retention using archival data
Eckles, J. E., & Stradley, E. G. (2011) · Social Psychology of Education · doi:10.1007/s11218-011-9173-z · Cited by 82 · see citing works
A Convolution-LSTM-Based Deep Neural Network for Cross-Domain MOOC Forum Post Classification
Wei, X., Lin, H., Yang, L., & Yu, Y. (2017) · Information · doi:10.3390/info8030092 · Cited by 81 · see citing works
Analyzing Student Work Patterns Using Programming Exercise Data
Spacco, J. W., Denny, P. C., Richards, B., Babcock, D., Hovemeyer, D., Moscola, J., & Duvall, R. C. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677297 · Cited by 75 · see citing works
A time series interaction analysis method for building predictive models of learners using log data
Brooks, C. A., Thompson, C. D. S., & Teasley, S. D. (2015) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2723576.2723581 · Cited by 75 · see citing works
Sinha, T., Li, N., Jermann, P., & Dillenbourg, P. (2014) · doi:10.3115/v1/w14-4108 · Cited by 74 · see citing works
Predicting Attrition Along the Way: The UIUC Model
Amnueypornsakul, B., Bhat, S., & Chinprutthiwong, P. (2014) · doi:10.3115/v1/w14-4110 · Cited by 50 · see citing works
Zafra, A., & Ventura, S. (2012) · Applied Soft Computing · doi:10.1016/j.asoc.2012.03.054 · Cited by 45 · see citing works
Longitudinal engagement, performance, and social connectivity
Zhu, M., Bergner, Y., Zhang, Y., et al. (2016) · International Conference on Learning Analytics & Knowledge (LAK) · doi:10.1145/2883851.2883934 · Cited by 40 · see citing works
Studying MOOC completion at scale using the MOOC replication framework
Andres, J. M. L., Baker, R. S., Gašević, D., Siemens, G., Crossley, S. A., & Joksimovíc, S. (2018) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3170358.3170369 · Cited by 38 · see citing works
Chen, Y., & Zhang, M. (2017) · ACM Turing Celebration Conference - China (ACM TURC) · doi:10.1145/3063955.3063959 · Cited by 34 · see citing works
Hammond, M. (1998) · Technology Pedagogy and Education · doi:10.1080/14759399800200041 · Cited by 31 · see citing works
Exploring N-gram Features in Clickstream Data for MOOC Learning Achievement Prediction
Li, X., Wang, T., & Wang, H. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-55705-2_26 · Cited by 26 · see citing works
Fostering a Learning Community in MOOCs.
Hmelo‐Silver, C. E., Rosé, C. P., & Levy, J. (2014) · Cited by 10 · see citing works
A multiple-view analysis model of debugging processes
Uchida, S., Monden, A., Iida, H., Matsumoto, K., & Kudo, H. (2003) · International Symposium on Empirical Software Engineering · doi:10.1109/isese.2002.1166933 · Cited by 6 · see citing works
Sinha, T., Li, N., Jermann, P., et al. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.5887 · Cited by 5 · see citing works
Stopout prediction in massive open online courses
Taylor, C. (2014) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 16
- [10][10] C. Brooks, C. Thompson, and S. Teasley. A time series interaction analysis method for building predictive models of learners using log data. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge , LAK '15, pages 126-135, New York, NY, USA, 2015. ACM. -
- [11][11] R. Brown, C. Lynch, M. Eagle, J. Albert, T. Barnes, R. S. Baker, Y. Bergner, and D. S. McNamara. Good communities and bad communities: Does membership affect performance? In Proceedings of the 8th International Conference on Educational Data Mining, EDM 2015, Madrid, Spain, June 26-29, 2015 , pages 612-613, 2015. -
- [12][12] R. Brown, C. Lynch, Y. Wang, M. Eagle, J. Albert, T. Barnes, R. S. Baker, Y. Bergner, and D. S. McNamara. Communities of performance & communities of preference. In EDM (Workshops) , 2015. -
- [21][21] M. Fisher. Using lotus notes learning space for staff development in public schools. Journal of Interactive Learning Research , 9(3):221, 1998. -
- [24][24] N. Gitinabard, F. Khoshnevisan, C. F. Lynch, and E. Y. Wang. Your actions or your associates? predicting certification and dropout in moocs with behavioral and social features. In International Conference on Educational Data Mining (EDM) , pages 404-410, 2018. -
- [25][25] N. Gitinabard, L. Xue, C. Lynch, S. Heckman, and T. Barnes. A social network analysis on blended courses. In Proceedings of the 10th International Conference on Educational Data Mining, EDM 2017(Workshops), Wuhan, China, June 25-28, 2017 , 2017. -
- [28][28] J. He, J. Bailey, B. I. Rubinstein, and R. Zhang. Identifying at-risk students in massive open online courses. In AAAI , pages 17491755, 2015. -
- [31][31] Z. Ibrahim and D. Rusli. Predicting students' academic performance: comparing artificial neural network, decision tree and linear regression. In 21st Annual SAS Malaysia Forum , pages 16, 2007. -
- [32][32] S. Jiang, S. M. Fitzhugh, and M. Warschauer. Social positioning and performance in moocs. In Proceedings of the Workshops held at Educational Data Mining 2014, co-located with 7th International Conference on Educational Data Mining (EDM 2014) , volume 1183, page 14, 2014. -
- [33][33] S. Jiang, S. M. Fitzhugh, and M. Warschauer. Social positioning and performance in moocs. In Workshop on Graph-Based Educational Data Mining , volume 14, 2014. -
- [45][45] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011. -
- [47][47] A. Ramesh, D. Goldwasser, B. Huang, H. Daum´ e III, and L. Getoor. Modeling learner engagement in moocs using probabilistic soft logic. In NIPS Workshop on Data Driven Education , volume 21, page 62, 2013. -
- [50][50] A. Sheshadri, N. Gitinabard, C. F. Lynch, T. Barnes, and S. Heckman. Predicting student performance based on online study habits: A study of blended courses. In Proceedings of the 11th International Conference on Educational Data Mining (EDM) 2018, Buffalo, US , pages 411-417, 2018. -
- [54][54] C. Tucker, B. K. Pursel, and A. Divinsky. Mining student-generated textual data in moocs and quantifying their effects on student performance and learning outcomes. Computers in Education Journal , 5(4):84-95, 2014. -
- [61][61] D. Yang, R. Kraut, and C. P. Ros´ e. Exploring the effect of student confusion in massive open online courses. Journal of Educational Data Mining , 8(1), 2016. -
- [62][62] D. Yang, T. Sinha, D. Adamson, and C. P. Ros´ e. Turn on, tune in, drop out: Anticipating student dropouts in massive open online courses. In Proceedings of the 2013 NIPS Data-driven education workshop , volume 11, page 14, 2013. -
Cited by
Sorted by citations
Proactive and reactive engagement of artificial intelligence methods for education: a review
Mallik, S., & Gangopadhyay, A. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1151391 · Cited by 139 · see citing works
- Authors
- Gitinabard, N., Xu, Y., Heckman, S., Barnes, T., & Lynch, C. F.
- Affiliations
- North Carolina State University (US)
- Venue
- IEEE Transactions on Learning Technologies
- Published
- 2019-04-01
- Volume
- 12
- Issue
- 2
- Pages
- 184-197
- ISSN
- 1939-1382 (ISSN-L)
- 2372-0050
- OpenAlex
- W2935686048
- Cited by
- 60
- References
- 41
- Citations by year
- 2026: 6 · 2025: 6 · 2024: 11 · 2023: 6 · 2022: 9 · 2021: 16 · 2020: 5 · 2019: 1
- FWCI
- 8.39
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Learning Technologies (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1821475)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Education › Online and Blended Learning
- Keywords
- blended learning, student performance prediction, model generalization, study habits, social interaction, early prediction, data requirements, student support interventions, secondary education, online learning platforms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
