MOOCs Meet Measurement Theory: A Topic-Modelling Approach
Open accessAbstract
This paper adapts topic models to the psychometric testing of MOOC students based on their online forum postings. Measurement theory from education and psychology provides statistical models for quantifying a person's attainment of intangible attributes such as attitudes, abilities or intelligence. Such models infer latent skill levels by relating them to individuals' observed responses on a series of items such as quiz questions. The set of items can be used to measure a latent skill if individuals' responses on them conform to a Guttman scale. Such well-scaled items differentiate between individuals and inferred levels span the entire range from most basic to the advanced. In practice, education researchers manually devise items (quiz questions) while optimising well-scaled conformance. Due to the costly nature and expert requirements of this process, psychometric testing has found limited use in everyday teaching. We aim to develop usable measurement models for highly-instrumented MOOC delivery platforms, by using participation in automatically-extracted online forum topics as items. The challenge is to formalise the Guttman scale educational constraint and incorporate it into topic models. To favour topics that automatically conform to a Guttman scale, we introduce a novel regularisation into non-negative matrix factorisation-based topic modelling. We demonstrate the suitability of our approach with both quantitative experiments on three Coursera MOOCs, and with a qualitative survey of topic interpretability on two MOOCs by domain expert interviews.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
It is possible to apply topic-modelling-based machine learning approaches to infer learners' attitude, ability and latent skill levels based on forum discussion data if the forum is actively used.
“It is possible to apply emerging topic-modelling-based machine learning approaches to infer learners’ attitude, ability and latent skill levels based on forum discussion data if the forum is actively used by learners.28”
From Towards AI-Powered Personalization in MOOC Learning (2017) · cites this paper as [28]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Learning the parts of objects by non-negative matrix factorization
Lee, D. D., & Seung, H. S. (1999) · Nature · doi:10.1038/44565 · Cited by 13,799 · see citing works
ALGORITHMS FOR NON-NEGATIVE MATRIX FACTORIZATION
Lee, D. D., & Seung, H. S. (2000) · Cited by 7,972 · see citing works
ALGORITHMS FOR NON-NEGATIVE MATRIX FACTORIZATION
Seung, D., & Lee, L.-W. (2001) · Cited by 4,753 · see citing works
Measurement, Design, and Analysis: An Integrated Approach.
Funk, S. G., Pedhazur, E. J., & Schmelkin, L. P. (1992) · Journal of the American Statistical Association · doi:10.2307/2290248 · Cited by 4,007 · see citing works
The theory and practice of item response theory.
Ayala, D. (2009) · Guilford Press eBooks · Cited by 1,812 · see citing works
Deconstructing disengagement: analyzing learner subpopulations in massive open online courses
Kizilcec, R. F., Piech, C., & Schneider, E. (2013) · Proceedings of the Third International Conference on Learning Analytics and Knowledge · doi:10.1145/2460296.2460330 · Cited by 984 · see citing works
Engaging with massive online courses
Anderson, A., Huttenlocher, D. P., Kleinberg, J. M., & Leskovec, J. (2014) · International World Wide Web Conference (WWW) · doi:10.1145/2566486.2568042 · Cited by 524 · see citing works
“Predicting MOOC dropout over weeks using machine learning methods,”
Kloft, M., Stiehler, F., Zheng, Z., & Pinkwart, N. (2014) · Proceedings of the EMNLP 2014 Workshop on Analysis of Large Scale Social Interaction in MOOCs · doi:10.3115/v1/w14-4111 · Cited by 325 · see citing works
Beaudoin, M. F. (2002) · The Internet and Higher Education · doi:10.1016/s1096-7516(02)00086-6 · Cited by 311 · see citing works
Sentiment Analysis in MOOC Discussion Forums: What does it tell us?
Wen, M., Yang, D., & Rosé, C. P. (2014) · Cited by 285 · see citing works
Tuned Models of Peer Assessment in MOOCs
Piech, C., Huang, J., Chen, Z., Do, C., Ng, A., & Koller, D. (2013) · arXiv · doi:10.48550/arxiv.1307.2579 · Cited by 262 · see citing works
Dropout Prediction in MOOCs using Learner Activity Features
Halawa, S. A., Greene, D., & Mitchell, J. (2014) · E-learning papers · Cited by 227 · see citing works
The Basis for Scalogram Analysis
Guttman, L. (2017) · Routledge eBooks · doi:10.4324/9781315128948-16 · Cited by 222 · see citing works
Identifying At-Risk Students in Massive Open Online Courses
He, J., Bailey, J., Rubinstein, B. I. P., & Zhang, R. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9471 · Cited by 215 · see citing works
Self-citationBinary Matrix Factorization with Applications
Zhang, Z., Li, T., Ding, C. H. Q., & Zhang, X. (2007) · Proceedings · doi:10.1109/icdm.2007.99 · Cited by 161 · see citing works
Exploring the Effect of Confusion in Discussion Forums of Massive Open Online Courses
Yang, D., Wen, M.-M., Howley, I., Kraut, R. E., & Rosé, C. (2015) · ACM Conference on Learning @ Scale · doi:10.1145/2724660.2724677 · Cited by 137 · see citing works
“Learning latent engagement patterns of students in online courses,”
Ramesh, A., Goldwasser, D., Huang, B., Daume III, H., & Getoor, L. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8920 · Cited by 123 · see citing works
Binary matrix factorization for analyzing gene expression data
Zhang, Z., Li, T., Ding, C. H. Q., Ren, X., & Zhang, X. (2009) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-009-0145-2 · Cited by 103 · see citing works
Understanding MOOC Discussion Forums using Seeded LDA
Ramesh, A., Goldwasser, D., Huang, B., Daumé, H., & Getoor, L. C. (2014) · Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.3115/v1/w14-1804 · Cited by 95 · see citing works
Weakly Supervised Models of Aspect-Sentiment for Online Course Discussion Forums
Ramesh, A., Kumar, S. H., Foulds, J. R., & Getoor, L. C. (2015) · doi:10.3115/v1/p15-1008 · Cited by 66 · see citing works
Question recommendation with constraints for massive open online courses
Yang, D., Adamson, G. D., & Rosé, C. P. (2014) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2645710.2645748 · Cited by 65 · see citing works
Engaging with Massive Online CoursesPreprint
Anderson, A., Huttenlocher, D. P., Kleinberg, J. M., & Leskovec, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1403.3100 · Cited by 58 · see citing works
Online Courses Recommendation based on LDA.
Guzman, R., Cervantes, E. V., Quispe, L. V. C., & Ochoa-Luna, J. E. (2014) · Cited by 53 · see citing works
Measurement, design, and analysis: An integrated approach, Student ed.
Pedhazur, E. J., & Schmelkin, L. P. (1991) · Cited by 51 · see citing works
Uncovering hidden engagement patterns for predicting learner performance in MOOCs
Ramesh, A., Goldwasser, D., Huang, B., Daumé, H., & Getoor, L. C. (2014) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2556325.2567857 · Cited by 36 · see citing works
Probabilistic Graphical Models for Boosting Cardinal and Ordinal Peer Grading in MOOCs
Mi, F., & Yeung, D.-Y. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9210 · Cited by 34 · see citing works
Study Partners Recommendation for xMOOCs Learners
Bin, X., & di Yang, D. (2015) · Computational Intelligence and Neuroscience · doi:10.1155/2015/832093 · Cited by 26 · see citing works
Communication Communities in MOOCsPreprint
Gillani, N., Eynon, R., Osborne, M. A., Hjorth, I., & Roberts, S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1403.4640 · Cited by 18 · see citing works
Crowd-sourced learning in MOOCs
Milligan, S. (2015) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2723576.2723596 · Cited by 16 · see citing works
Self-citationExploring Class Discussions from a Massive Open Online Course (MOOC) on Cartography
Robinson, A. C. (2014) · Lecture notes in geoinformation and cartography · doi:10.1007/978-3-319-07926-4_14 · Cited by 6 · see citing works
MOOCs Meet Measurement Theory: A Topic-Modelling ApproachPreprint
He, J., Rubinstein, B. I. P., Bailey, J., et al. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.07961 · Cited by 2 · see citing works
Self-citationCrowd-sourced learning in MOOCs: Learning analytics meets measurement theory
Milligan, S. (2015) · ACM International Conference Proceeding Series · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
1 of the 33 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Abdi, H. 2010. Guttman scaling. In Salkind, N. J., ed., Encyclopedia of Research Design . SAGE Publications.
- Dıez, J.; Luaces, O.; Alonso-Betanzos, A.; Troncoso, A.; and Bahamonde, A. 2013. Peer assessment in MOOCs using preference learning via matrix factorization. In NIPS Workshop on Data Driven Education .
- Lee, D. D., and Seung, H. S. 2001. Algorithms for nonnegative matrix factorization. In Advances in Neural Information Processing Systems , 556-562.
- Yang, D.; Sinha, T.; Adamson, D.; and Rose, C. P. 2013. Turn on, tune in, drop out: Anticipating student dropouts in massive open online courses. In Proceedings of the 2013 NIPS Data-Driven Education Workshop , volume 11, 14.
Cited by
Sorted by citations
Towards AI-Powered Personalization in MOOC Learning
Yu, H., Miao, C., Leung, C., & White, T. J. (2017) · npj Science of Learning · doi:10.1038/s41539-017-0016-3 · Cited by 131 · see citing works
McInerney, F., Doherty, K., Bindoff, A. D., Robinson, A., & Vickers, J. C. (2017) · Palliative Medicine · doi:10.1177/0269216317743433 · Cited by 63 · see citing works
Artificial Intelligence for Social Good: A SurveyPreprint
Shi, Z. R., Wang, C., & Fang, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.01818 · Cited by 108 · see citing works
Artificial Intelligence Powered MOOCs: A Brief Survey
Fauvel, S., Yu, H., Miao, C., Cui, L., Song, H., Zhang, L., Li, X., & Leung, C. (2018) · IEEE International Conference on Agents (ICA) · doi:10.1109/agents.2018.8460059 · Cited by 20 · see citing works
Methodological foundations for the measurement of learning in learning analytics
Milligan, S. (2018) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3170358.3170391 · Cited by 13 · see citing works
Self-citationStandards for Developing Assessments of Learning Using Process Data
Milligan, S. (2020) · The enabling power of assessment · doi:10.1007/978-3-030-41956-1_13 · Cited by 12 · see citing works
Self-citationHe, Z., Xia, W., Dong, K., Guo, H., Tang, R., Xia, D., & Zhang, R. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539107 · Cited by 11 · see citing works
Self-citationMilligan, S., He, J., Bailey, J., Zhang, R., & Rubinstein, B. I. P. (2016) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/2883851.2883956 · Cited by 4 · see citing works
Self-citationTopicResponse: A Marriage of Topic Modelling and Rasch Modelling for Automatic Measurement in MOOCsPreprint
He, J., Rubinstein, B. I. P., Bailey, J., Zhang, R., & Milligan, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.08720
Self-citation
- Authors
- He, J., Rubinstein, B., Bailey, J., Zhang, R., Milligan, S., & Chan, J.
- Affiliations
- The University of Melbourne (AU)
- RMIT University (AU)
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2016-02-21
- Volume
- 30
- Issue
- 1
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W2963372297
- Cited by
- 10
- References
- 33
- Citations by year
- 2022: 1 · 2020: 2 · 2018: 2 · 2017: 2 · 2016: 3
- FWCI
- 1.86
- Type
- conference paper
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
- RMIT Research Repository (RMIT University Library) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- MOOCs, psychometric assessment, topic modeling, Guttman scale, non-negative matrix factorization, regularization, item generation, Coursera, topic interpretability
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
