The DOI matches its Crossref record; 70 of the 71 references it cites were found in a registry or an index.
Evaluating the Quality of Learning Resources: A Learnersourcing Approach
Abstract
Learnersourcing is emerging as a viable approach for mobilizing the learner community and harnessing the intelligence of learners as creators of learning resources. Previous works have demonstrated that the quality of resources developed by students is quite diverse with some resources meeting rigorous judgmental criteria, whereas other resources are ineffective, inappropriate, or incorrect. Consequently, to effectively utilize these large repositories of resources in student learning, there is a need for a selection and moderation process to separate high-quality resources from low-quality ones in such repositories. Instructors and domain experts are potentially the most reliable source for doing this task; however, their availability is often quite limited. This article explores whether and how learnersourcing, as an alternative approach, can be used for evaluating the quality of learning resources. To do so, we first follow a data-driven approach to explore students' ability in judging the quality of learning resources. Results from this study suggest that, overall, ratings provided by students strongly correlate with ratings from experts; however, students' ability in evaluating learning resources can also vary significantly. We then present a consensus approach based on matrix factorization and indicate how it can be used for improving the accuracy of aggregating learnersourced decisions. In this article, we also demonstrate how utilizing information on student performance and incorporating ratings from domain experts on a limited number of learning resources can be leveraged to further improve the accuracy of the results.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“the judgements of students as experts-in-training cannot wholly be trusted, which compromises the reliability of peer assessment as an assessment instrument.”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Abdi et al., 2021
In the citing paper
“However, summary statistics suffer from the assumption that all students have a similar judgmental ability, which has proven incorrect (Abdi et al., 2021).”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Abdi et al., 2021
Citations
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Taxonomy of Educational Objectives. Handbook I: Cognitive Domain
Bloom, B., & Krathwohl, D. R. (1966) · Cited by 4,630
Factorization meets the neighborhood: a multifaceted collaborative filtering model
Koren, Y. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401944 · Cited by 3,852
Taxonomy of Educational Objectives, Handbook I: Cognitive Domain
(2010) · Encyclopedia of Curriculum Studies · doi:10.4135/9781412958806.n446 · Cited by 2,317
(1996) · Cited by 1,549
Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise
Whitehill, J., Wu, T., Bergsma, J., Movellan, J. R., & Ruvolo, P. (2009) · Cited by 1,004
Raykar, V. C., Yu, S., Zhao, L. H., Valadez, G. H., Florin, C., Bogoni, L., & Moy, L. P. (2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859894 · Cited by 951
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906
Addressing the Challenges of Inquiry-Based Learning Through Technology and Curriculum Design
Edelson, D. C., Gordin, D. N., & Pea, R. D. (1999) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.1207/s15327809jls0803 · Cited by 875
Developing evaluative judgement: enabling students to make decisions about the quality of work
Tai, J., Ajjawi, R., Boud, D., Dawson, P., & Panadero, E. (2018) · Higher Education · doi:10.1007/s10734-017-0220-3 · Cited by 810
Research works printed but not linked: 2
- wind, 2018, Peer feedback with peergrade, Proc 13th Int Conf e-Learning, 184
- wang, 2011, Managing crowdsourcing workers, Proc Winter Conf Bus Intell, 10
Cited by
Assessment in the age of artificial intelligence
Swiecki, Z., Khosravi, H., Chen, G., Martinez-Maldonado, R., Lodge, J. M., Milligan, S., Selwyn, N., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100075 · Cited by 409
Self-citationComparing Code Explanations Created by Students and Large Language Models
Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., & Hellas, A. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588785 · Cited by 210
E-Learning Course Recommender System Using Collaborative Filtering Models
Jena, K. K., Bhoi, S. K., Malik, T. K., Sahoo, K. S., Jhanjhi, N. Z., Bhatia, S., & Amsaad, F. (2022) · Electronics · doi:10.3390/electronics12010157 · Cited by 184
Denny, P. C., Khosravi, H., Hellas, A., Leinonen, J., & Sarsa, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.10509 · Cited by 118
Self-citationEnhancing peer assessment with artificial intelligence
Topping, K. J., Gehringer, E. F., Khosravi, H., Gudipati, S., Jadhav, K., & Susarla, S. (2025) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00501-1 · Cited by 104
Self-citation
Metadata
- Authors
- Abdi, S., Khosravi, H., Sadiq, S., & Demartini, G.
- Affiliations
- The University of Queensland (AU)
- Venue
- IEEE Transactions on Learning Technologies
- Published
- 2021-02
- Volume
- 14
- Issue
- 1
- Pages
- 81-92
- ISSN
- 1939-1382 (ISSN-L)
- 2372-0050
- OpenAlex
- W3129460529
- Cited by
- 40
- References
- 71
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 8 · 2023: 7 · 2022: 9 · 2021: 5
- FWCI
- 6.14
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Learning Technologies (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Education › Open Education and E-Learning
- Keywords
- expert ratings, consensus aggregation, matrix factorization, student performance data, community of learners
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating the Quality of Learning Resources: A Learnersourcing Approach. https://app.cochat.ai/papers/10.1109/tlt.2021.3058644
- The paper
- Abdi, S., Khosravi, H., Sadiq, S., & Demartini, G. (2021). Evaluating the Quality of Learning Resources: A Learnersourcing Approach. IEEE Transactions on Learning Technologies. https://doi.org/10.1109/tlt.2021.3058644
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.