Evaluating the Quality of Learning Resources: A Learnersourcing Approach
The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
Learnersourcing is emerging as a viable approach for mobilizing the learner community and harnessing the intelligence of learners as creators of learning resources. Previous works have demonstrated that the quality of resources developed by students is quite diverse with some resources meeting rigorous judgmental criteria, whereas other resources are ineffective, inappropriate, or incorrect. Consequently, to effectively utilize these large repositories of resources in student learning, there is a need for a selection and moderation process to separate high-quality resources from low-quality ones in such repositories. Instructors and domain experts are potentially the most reliable source for doing this task; however, their availability is often quite limited. This article explores whether and how learnersourcing, as an alternative approach, can be used for evaluating the quality of learning resources. To do so, we first follow a data-driven approach to explore students' ability in judging the quality of learning resources. Results from this study suggest that, overall, ratings provided by students strongly correlate with ratings from experts; however, students' ability in evaluating learning resources can also vary significantly. We then present a consensus approach based on matrix factorization and indicate how it can be used for improving the accuracy of aggregating learnersourced decisions. In this article, we also demonstrate how utilizing information on student performance and incorporating ratings from domain experts on a limited number of learning resources can be leveraged to further improve the accuracy of the results.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“the judgements of students as experts-in-training cannot wholly be trusted, which compromises the reliability of peer assessment as an assessment instrument.”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Abdi et al., 2021
In the citing paper
“However, summary statistics suffer from the assumption that all students have a similar judgmental ability, which has proven incorrect (Abdi et al., 2021).”
From Assessment in the age of artificial intelligence (2022) · cites this paper as Abdi et al., 2021
Citations
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532 · cited 1×
“The result of the interrater agreement among domain experts using the intraclass correlation coefficient (ICC) [62] suggests an excellent agreement among them for evaluating the quality of learning resources (ICC[3, k ] = 0.84]).”
MethodInfluentialTaxonomy of Educational Objectives. Handbook I: Cognitive Domain
Bloom, B., & Krathwohl, D. R. (1966) · Cited by 4,631
Factorization meets the neighborhood: a multifaceted collaborative filtering model
Koren, Y. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401944 · Cited by 3,852 · cited 1×
“In all analyses, we use an extension of MF called biased-MF proposed by Koren [63] that incorporates mean normalization and a bias parameter for each student and learning resource in ratings.”
MethodTaxonomy of Educational Objectives, Handbook I: Cognitive Domain
(2010) · Encyclopedia of Curriculum Studies · doi:10.4135/9781412958806.n446 · Cited by 2,317
(1996) · Cited by 1,549
Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise
Whitehill, J., Wu, T., Bergsma, J., Movellan, J. R., & Ruvolo, P. (2009) · neural information processing systems · Cited by 1,364
Raykar, V. C., Yu, S., Zhao, L. H., Valadez, G. H., Florin, C., Bogoni, L., & Moy, L. P. (2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859894 · Cited by 951
Designing Theory-Driven User-Centric Explainable AI
Wang, D., Yang, Q., Abdul, A., & Lim, B. Y. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300831 · Cited by 906 · cited 1×
“While some studies suggest that explainable AI (XIA) [67] is not always required [68], the use of black-box outcomes seems to be inadequate for educational settings where educators strive to enable students to develop their own vision, reasoning, and appreciation for inquiry and investigation, and…”
BackgroundAddressing the Challenges of Inquiry-Based Learning Through Technology and Curriculum Design
Edelson, D. C., Gordin, D. N., & Pea, R. D. (1999) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.1207/s15327809jls0803 · Cited by 875
Developing evaluative judgement: enabling students to make decisions about the quality of work
Tai, J., Ajjawi, R., Boud, D., Dawson, P., & Panadero, E. (2018) · Higher Education · doi:10.1007/s10734-017-0220-3 · Cited by 810 · cited 1×
“Furthermore, it has the potential to help with developing the evaluative judgment of students, which is an important aspect of the learning process [25]–[27].”
BackgroundSoylent: a word processor with a crowd inside
Bernstein, M. S., Little, G., Miller, R. C., Hartmann, B., Ackerman, M. S., Karger, D. R., Crowell, D., & Panovich, K. (2010) · Proceedings of the 23nd annual ACM symposium on User interface software and technology · doi:10.1145/1866029.1866078 · Cited by 686 · cited 1×
“Traditional approaches for crowdsourcing consensus use general statistical aggregations, such as the arithmetic mean, median, or majority vote [38].”
MethodRaskin, J. D. (2002) · Cited by 492
Demartini, G., Difallah, D. E., & Cudré-Mauroux, P. (2012) · Proceedings of the 21st international conference on World Wide Web · doi:10.1145/2187836.2187900 · Cited by 448 · cited 1×
“Most of the state-of-the-art consensus approaches developed within crowdsourcing community rely on black-box machine learning algorithms (e.g., [37], [65]) for aggregating crowd-sourced decisions.”
BackgroundSelf-citationTruth inference in crowdsourcing: is the problem solved?
Zheng, Y., Li, G., Li, Y., Shan, C., & Cheng, R. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3055540.3055547 · Cited by 421 · cited 1×
“While designing mechanisms for accurate integration of crowdsourced data through machine learning algorithms has been studied extensively within the crowdsourcing community [28], the use of these approaches for aggregating learnersourced decisions has received little attention.”
BackgroundMyMediaLite: a free recommender system library
Gantner, Z., Rendle, S., Freudenthaler, C., & Schmidt-Thieme, L. (2011) · Proceedings of the fifth ACM conference on Recommender systems · doi:10.1145/2043932.2043989 · Cited by 368 · cited 1×
“of MyMediaLite [64] for all of our conducted studies.”
ResultAddressing the challenges of inquiry‐based learning through technology and curriculum design
Edelson, D., Gordin, D., & Pea, R. (1999) · Journal of the Learning Sciences · doi:10.1207/s15327809jls0803&4_3 · Cited by 366
How students learn: Reforming schools through learner-centered education.
(1998) · American Psychological Association eBooks · doi:10.1037/10258-000 · Cited by 330 · cited 1×
“This approach, as Hill deliberates [7], originates from the student as producer model [8], which itself builds on the existing literature on student-centered learning [9], inquiry-based learning [10], and contributing student pedagogy [11].”
BackgroundProgrammatic gold: targeted and scalable quality assurance in crowdsourcing
Oleson, D., Sorokin, A. A., Laughlin, G., Hester, V., Le, J., & Biewald, L. (2011) · National Conference on Artificial Intelligence · Cited by 259
Community-based bayesian aggregation models for crowdsourcing
Venanzi, M., Guiver, J., Kazai, G., Kohli, P., & Shokouhi, M. (2014) · Proceedings of the 23rd international conference on World wide web · doi:10.1145/2566486.2567989 · Cited by 242 · cited 4×
“With the widespread adoption of crowdsourcing in the problems that require human intelligence, optimal integration of crowdsourced decisions in the absence of ground truth has become a challenging task [37].”
BackgroundInfluentialPeerWise: students sharing their multiple choice questions
Denny, P., Hamer, J., Luxton-Reilly, A., & Purchase, H. (2008) · Proceedings of the Fourth international Workshop on Computing Education Research · doi:10.1145/1404520.1404526 · Cited by 211 · cited 3×
“Previous studies have shown that students can create repositories of high-quality learning resources that meet rigorous judg-mental criteria [16], [20], [23].”
BackgroundBull, S., & Kay, J. (2010) · Studies in Computational Intelligence · doi:10.1007/978-3-642-14363-2_15 · Cited by 209 · cited 1×
“Much of the existing works on the need for open and XIA models in education has focused on open learner models [69], educational recommender systems [70], and learning analytics dash-boards [71].”
BackgroundOn-line question-posing and peer-assessment as means for web-based knowledge sharing in learning
Barak, M., & Rafaeli, S. (2004) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2003.12.005 · Cited by 175 · cited 3×
“Barak and Rafaeli [14] conducted a study on an MBA course, in which students enrolled in the course were responsible for contributing learning resources to an online repository as well as ranking the resources contributed by other students to the repository.”
BackgroundData-driven interaction techniques for improving navigation of educational videos
Kim, J., Guo, P. J., Cai, C. J., Li, S.-W. (., Gajos, K. Z., & Miller, R. C. (2014) · Proceedings of the 27th annual ACM symposium on User interface software and technology · doi:10.1145/2642918.2647389 · Cited by 166 · cited 1×
“Successful examples of learnersourced artifacts created by students include traces of videos watched by students [2], annotating videos for future students [1], creating open-ended artifact, such as solutions [3], or explanations [4], [5], and curriculum design [6].”
BackgroundInstrumenting the crowd: using implicit behavioral measures to predict task performance
Rzeszotarski, J. M., & Kittur, A. (2011) · Proceedings of the 24th annual ACM symposium on User interface software and technology · doi:10.1145/2047196.2047199 · Cited by 164
Catalytic assessment: understanding how MCQs and EVS can foster deep learning
Draper, S. W. (2009) · British Journal of Educational Technology · doi:10.1111/j.1467-8535.2008.00920.x · Cited by 157 · cited 1×
“Previous studies have reported that placing the responsibility of content creation in the hands of students reinforces and deepens students’ understanding of the course content through engaging them in cognitively demanding tasks [14], [15].”
BackgroundHamer, J., Cutts, Q., Jackova, J., Luxton-Reilly, A., McCartney, R., Purchase, H., Riedesel, C., Saeli, M., Sanders, K., & Sheard, J. (2008) · ACM SIGCSE Bulletin · doi:10.1145/1473195.1473242 · Cited by 155 · cited 3×
“This approach, as Hill deliberates [7], originates from the student as producer model [8], which itself builds on the existing literature on student-centered learning [9], inquiry-based learning [10], and contributing student pedagogy [11].”
BackgroundAXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning
Williams, J. J., Kim, J., Rafferty, A., Maldonado, S., Gajos, K. Z., Lasecki, W. S., & Heffernan, N. (2016) · Proceedings of the Third (2016) ACM Conference on Learning @ Scale · doi:10.1145/2876034.2876042 · Cited by 143 · cited 2×
“Successful examples of learnersourced artifacts created by students include traces of videos watched by students [2], annotating videos for future students [1], creating open-ended artifact, such as solutions [3], or explanations [4], [5], and curriculum design [6].”
BackgroundMethodA web‐based learning system for question‐posing and peer assessment
Yu, F., Liu, Y., & Chan, T. (2005) · Innovations in Education and Teaching International · doi:10.1080/14703290500062557 · Cited by 140 · cited 2×
“Yu et al. [30] reported on a web-based online learning system that enables students to contribute learning resources to the repository of the system.”
BackgroundBasic Concepts in Item and Test Analysis.
Matlock‐Hetzel, S. (1997) · Cited by 120
Paré, D. E., & Joordens, S. (2008) · Journal of Computer Assisted Learning · doi:10.1111/j.1365-2729.2008.00290.x · Cited by 113 · cited 1×
“For example, many peer grading systems, such as Mechanical TA [52], Peer Assessment [53], Peergrade [54], Arop € a [55], and peerScholar [56], use statistical aggregations.”
BackgroundThe PeerWise system of student contributed assessment questions
Denny, P. C., Luxton-Reilly, A., & Hamer, J. (2008) · doi:10.5555/1379249.1379255 · Cited by 110 · cited 5×
“For example, it can be used by students for studying [21], by instructors for creating exams or assignments [20], and by adaptive learning systems for recommending personalized learning resources [22].”
BackgroundInfluentialAre explanations always important?: a study of deployed, low-cost intelligent interactive systems
Bunt, A., Lount, M., & Lauzon, C. (2012) · Proceedings of the 2012 ACM international conference on Intelligent User Interfaces · doi:10.1145/2166966.2166996 · Cited by 106 · cited 1×
“While some studies suggest that explainable AI (XIA) [67] is not always required [68], the use of black-box outcomes seems to be inadequate for educational settings where educators strive to enable students to develop their own vision, reasoning, and appreciation for inquiry and investigation, and…”
BackgroundDevelopment and Adoption of an Adaptive Learning System: Reflections and Lessons Learned
Khosravi, H., Sadiq, S., & Gasevic, D. (2020) · Proceedings of the 51st ACM Technical Symposium on Computer Science Education · doi:10.1145/3328778.3366900 · Cited by 104 · cited 1×
“At its core, RiPPLE is an adaptive educational system that dynamically adjusts the level or type of instruction based on an individual student’s ability or preferences to provide a customized learning experience [58].”
BackgroundSelf-citationBates, S. P., Galloway, R. K., McBride, K. L., Rebello, N. S., Engelhardt, P. V., & Singh, C. (2012) · AIP Conference Proceedings · doi:10.1063/1.3680009 · Cited by 85 · cited 1×
“…work in a clear and logical fashion [16], encourages them to reflect on the course objectives [12], enhances their conceptual understanding [17], and facilitates the capacity for students to relate their learning to their personal experiences, which is the core principle of constructivist…”
BackgroundRiPPLE: A Crowdsourced Adaptive Platform for Recommendation of Learning Activities
Khosravi, H., Kitto, K., & Williams, J. J. (2019) · arXiv · doi:10.48550/arxiv.1910.05522 · Cited by 81 · cited 6×
“1) Tool: This article uses a course-level, discipline-agnos-tic platform called RiPPLE [22].”
BackgroundMethodInfluentialSelf-citationRiPPLE: A Crowdsourced Adaptive Platform for Recommendation of Learning Activities
Khosravi, H., Kitto, K., & Williams, J. J. (2019) · Journal of Learning Analytics · doi:10.18608/jla.2019.63.12 · Cited by 78
Self-citationLearning to predict from crowdsourced data
Bi, W., Wang, L., Kwok, J. T.-Y., & Tu, Z. (2014) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 75
Student as producer: a pedagogy for the avant-garde?
Neary, M. (2010) · Lincoln Repository (University of Lincoln) · Cited by 71
Kurve, A., Miller, D. J., & Kesidis, G. (2015) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2014.2327026 · Cited by 69
Complementing educational recommender systems with open learner models
Abdi, S., Khosravi, H., Sadiq, S., & Gasevic, D. (2020) · Proceedings of the Tenth International Conference on Learning Analytics & Knowledge · doi:10.1145/3375462.3375520 · Cited by 68 · cited 1×
“Much of the existing works on the need for open and XIA models in education has focused on open learner models [69], educational recommender systems [70], and learning analytics dash-boards [71].”
BackgroundSelf-citationMedical students create multiple-choice questions for learning in pathology education: a pilot study
Grainger, R., Dai, W., Osborne, E., & Kenwright, D. (2018) · BMC Medical Education · doi:10.1186/s12909-018-1312-1 · Cited by 62 · cited 1×
“…analyses indicate that students who contribute to the creation of resources use higher order of thinking skills to generate good quality resources [32], as well as taking an active role in their learning [33], and that there is a positive association between students’ engagement in PeerWise and…”
BackgroundHuman Beyond the Machine: Challenges and Opportunities of Microtask Crowdsourcing
Gadiraju, U., Demartini, G., Kawase, R., & Dietze, S. (2015) · IEEE Intelligent Systems · doi:10.1109/mis.2015.66 · Cited by 54 · cited 1×
“However, previous studies showed that the quality of the crowdsourced data varies across different crowdsourcers and is affected by several factors, such as skill, underlying motivation, and level of expertise [37], [39], [40].”
BackgroundSelf-citationUpGrade: Sourcing Student Open-Ended Solutions to Create Scalable Learning Opportunities
Wang, X., Talluri, S. T., Rosé, C. P., & Koedinger, K. R. (2019) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3330430.3333614 · Cited by 53 · cited 1×
“Successful examples of learnersourced artifacts created by students include traces of videos watched by students [2], annotating videos for future students [1], creating open-ended artifact, such as solutions [3], or explanations [4], [5], and curriculum design [6].”
BackgroundAdaptive Social Learning Based on Crowdsourcing
Karataev, E., & Zadorozhny, V. (2016) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2016.2515097 · Cited by 52 · cited 1×
“…et al. [35] developed “crowdlearning” in which students collaboratively create learning resources for each other, and Karataev and Zadorozhny [36] proposed a framework that combines concepts of crowdsourcing, online social networks, and adaptive systems to provide personalized learning…”
BackgroundThe Future of Adaptive Learning: Does the Crowd Hold the Key?
Heffernan, N. T., Ostrow, K. S., Kelly, K., Selent, D., Van Inwegen, E. G., Xiong, X., & Williams, J. J. (2016) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-016-0094-z · Cited by 50 · cited 2×
“Successful examples of learnersourced artifacts created by students include traces of videos watched by students [2], annotating videos for future students [1], creating open-ended artifact, such as solutions [3], or explanations [4], [5], and curriculum design [6].”
BackgroundMethodImproving quality of crowdsourced labels via probabilistic matrix factorization
Jung, H. J., & Lease, M. (2012) · National Conference on Artificial Intelligence · Cited by 50
Walsh, J. L., Harris, B. H. L., Denny, P., & Smith, P. (2018) · Postgraduate Medical Journal · doi:10.1136/postgradmedj-2017-135018 · Cited by 49 · cited 1×
“Previous studies have shown that students can create repositories of high-quality learning resources that meet rigorous judg-mental criteria [16], [20], [23].”
Background(2009) · doi:10.5555/1862712.1862724 · Cited by 49
Mechanical TA: Partially Automated High-Stakes Peer Grading
Wright, J. R., Thornton, C., & Leyton-Brown, K. (2015) · Proceedings of the 46th ACM Technical Symposium on Computer Science Education · doi:10.1145/2676723.2677278 · Cited by 48 · cited 1×
“For example, many peer grading systems, such as Mechanical TA [52], Peer Assessment [53], Peergrade [54], Arop€a [55], and peerScholar [56], use statistical aggregations.”
MethodCrowdsourcing content creation in the classroom
Hills, T. T. (2015) · Journal of Computing in Higher Education · doi:10.1007/s12528-015-9089-2 · Cited by 46 · cited 3×
“For example, Hills [7] presented a conceptual framework for learnersourcing content creation in education.”
BackgroundA Multivariate Elo-based Learner Model for Adaptive Educational Systems
Abdi, S., Khosravi, H., Sadiq, S. W., & Gašević, D. (2019) · Monash University Research Portal (Monash University) · doi:10.48550/arxiv.1910.12581 · Cited by 41 · cited 1×
“The upper part contains an interactive visualization widget allowing students to view an abstract representation of their knowledge state based on a set of topics associated with a course using an open learner model, as outlined in [59].”
BackgroundSelf-citationRiPLE: Recommendation in Peer-Learning Environments Based on Knowledge Gaps and Interests
Khosravi, H., Cooper, K., & Kitto, K. (2017) · Journal of Educational Data Mining · doi:10.5281/zenodo.3554627 · Cited by 40
Self-citationStudent use of PeerWise: A multi‐institutional, multidisciplinary evaluation
Kay, A. E., Hardy, J., & Galloway, R. (2019) · British Journal of Educational Technology · doi:10.1111/bjet.12754 · Cited by 35 · cited 1×
“…who contribute to the creation of resources use higher order of thinking skills to generate good quality resources [32], as well as taking an active role in their learning [33], and that there is a positive association between students’ engagement in PeerWise and their academic achievements [34].”
BackgroundTravel Recommendation via Fusing Multi-Auxiliary Information into Matrix Factorization
Chen, L., Wu, Z., Cao, J., Zhu, G., & Ge, Y. (2020) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3372118 · Cited by 35 · cited 1×
“3) MF works well in the presence of auxiliary data [57], which enables us to explore the impact of having access to additional information, such as student performance.”
BackgroundPeerwise: replication study of a student-collaborative self-testing web service in a u.s. setting
Denny, P., Hanks, B., & Simon, B. (2010) · Proceedings of the 41st ACM technical symposium on Computer science education · doi:10.1145/1734263.1734407 · Cited by 32
The quality of a PeerWise MCQ repository
Purchase, H. C., Hamer, J., Denny, P. C., & Luxton-Reilly, A. (2010) · doi:10.5555/1862219.1862238 · Cited by 28
Peer-review in practice: eight years of Aropä
Purchase, H., & Hamer, J. (2018) · Assessment & Evaluation in Higher Education · doi:10.1080/02602938.2018.1435776 · Cited by 22 · cited 1×
“For example, many peer grading systems, such as Mechanical TA [52], Peer Assessment [53], Peergrade [54], Arop€a [55], and peerScholar [56], use statistical aggregations.”
MethodCrowdsourcing for assessment items to support adaptive learning
Tackett, S. A., Raymond, M. R., Desai, R., Haist, S. A., Morales, A., Gaglani, S. M., & Clyman, S. G. (2018) · Medical Teacher · doi:10.1080/0142159x.2018.1490704 · Cited by 22 · cited 4×
“Previous studies have shown that students can create repositories of high-quality learning resources that meet rigorous judg-mental criteria [16], [20], [23].”
BackgroundInfluentialMorise, H., Oyama, S., & Kurihara, M. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.04.044 · Cited by 21 · cited 1×
“…approaches are the ones that regard the aggregation problem as an information recommendation problem and utilize collaborative filtering recommendation techniques, such as item-based collaborative filtering, MF, and tensor factorization, to aggregate crowdsourced data [40], [49]–[51].”
MethodA Note on the Behavior of Majority Voting in Multi-Class Domains with Biased Annotators
Hernández-González, J., Inza, I., & Lozano, J. A. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2018.2845400 · Cited by 19 · cited 1×
“Probabilistic models and, in particular, expectation–maximization, which are generally developed in the context of categorization tasks, such as in [37], [44]–[48], are well-known examples of these class of approaches.”
BackgroundOptimal Spot-Checking for Improving Evaluation Accuracy of Peer Grading Systems
Wang, W., An, B., & Jiang, Y. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11336 · Cited by 19
Multi-label crowd consensus via joint matrix factorization
Tu, J., Yu, G., Domeniconi, C., Wang, J., Xiao, G., & Guo, M. (2019) · Knowledge and Information Systems · doi:10.1007/s10115-019-01386-7 · Cited by 19 · cited 3×
“MF can induce a latent feature vector for each student and each learning resource, which can be used for inferring student ratings on all learning resources [40], [49].”
BackgroundMethodAutomated insightful drill-down recommendations for learning analytics dashboards
Shabaninejad, S., Khosravi, H., Indulska, M., Bakharia, A., & Isaias, P. T. (2020) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3375462.3375539 · Cited by 19 · cited 1×
“Much of the existing works on the need for open and XIA models in education has focused on open learner models [69], educational recommender systems [70], and learning analytics dash-boards [71].”
BackgroundSelf-citationFarasat, A., Nikolaev, A. G., Miller, S. M., & Gopalsamy, R. (2017) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3051457.3053990 · Cited by 18 · cited 1×
“Farasat et al. [35] developed “crowdlearning” in which students collaboratively create learning resources for each other, and Karataev and Zadorozhny [36] proposed a framework that combines concepts of crowdsourcing, online social networks, and adaptive systems to provide personalized learning…”
BackgroundUtilising Learnersourcing to Inform Design Loop Adaptivity
Darvishi, A., Khosravi, H., & Sadiq, S. W. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-57717-9_24 · Cited by 18 · cited 1×
“However, generally, these machine-learning-based consensus approaches are not understandable and transparent in terms of how decisions from each individual are weighted and how the final decision was made [66].”
BackgroundSelf-citationPractical Peer Prediction for Peer Assessment
Shnayder, V., & Parkes, D. (2016) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v4i1.13285 · Cited by 17
Active Learning for Student Affect Detection
Yang, T.-Y., Baker, R. S., Studer, C., Heffernan, N. T., & Lan, A. (2019) · Repository for Publications and Research Data (ETH Zurich) · doi:10.3929/ethz-b-000461351 · Cited by 17
Khosravi, H., Gyamfi, G., Hanna, B. E., & Lodge, J. M. (2020) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3375462.3375532 · Cited by 17 · cited 1×
“Furthermore, it has the potential to help with developing the evaluative judgment of students, which is an important aspect of the learning process [25]–[27].”
BackgroundSelf-citationStudents sharing and evaluating MCQs in a large first year engineering course
Denny, P. C., Hamer, J., & Luxton-Reilly, A. (2009) · 20th Annual Conference for the Australasian Association for Engineering Education, 6-9 December 2009: Engineering the Curriculum · Cited by 7
Collaborative filtering and rating aggregation based on multicriteria rating
Morise, H., Oyama, S., & Kurihara, M. (2017) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata.2017.8258477 · Cited by 5
Research works printed but not linked: 2
- wind, 2018, Peer feedback with peergrade, Proc 13th Int Conf e-Learning, 184
- wang, 2011, Managing crowdsourcing workers, Proc Winter Conf Bus Intell, 10
Cited by
Assessment in the age of artificial intelligence
Swiecki, Z., Khosravi, H., Chen, G., Martinez-Maldonado, R., Lodge, J. M., Milligan, S., Selwyn, N., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100075 · Cited by 409
Self-citationComparing Code Explanations Created by Students and Large Language Models
Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., & Hellas, A. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588785 · Cited by 210
“While learnersourcing techniques could be used to generate code explanations efficiently by directly involving students in their creation [23, 34], there are known issues relating to quality when learning content is sourced from students [1, 9].”
BackgroundE-Learning Course Recommender System Using Collaborative Filtering Models
Jena, K. K., Bhoi, S. K., Malik, T. K., Sahoo, K. S., Jhanjhi, N. Z., Bhatia, S., & Amsaad, F. (2022) · Electronics · doi:10.3390/electronics12010157 · Cited by 184
Denny, P. C., Khosravi, H., Hellas, A., Leinonen, J., & Sarsa, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.10509 · Cited by 119
“Recent research involving the RiPPLE platform has explored the potential of learnersourcing as an alternative approach for evaluating the quality of learning resources, finding a strong correlation between student and expert ratings of artefacts while acknowledging variations in students’ evaluation skills [34].”
BackgroundSelf-citationEnhancing peer assessment with artificial intelligence
Topping, K. J., Gehringer, E. F., Khosravi, H., Gudipati, S., Jadhav, K., & Susarla, S. (2025) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00501-1 · Cited by 106
“RiPPLE actively engages students in the creation of diverse and innovative study resources, a process also known as learnersourcing (Khosravi et al., 2023).”
Self-citation
Metadata
- Authors
- Abdi, S., Khosravi, H., Sadiq, S., & Demartini, G.
- Affiliations
- The University of Queensland (AU)
- Venue
- IEEE Transactions on Learning Technologies
- Published
- 2021-02
- Volume
- 14
- Issue
- 1
- Pages
- 81-92
- ISSN
- 1939-1382 (ISSN-L)
- 2372-0050
- OpenAlex
- W3129460529
- Cited by
- 40
- References
- 71
- Citations by year
- 2026: 5 · 2025: 6 · 2024: 8 · 2023: 7 · 2022: 9 · 2021: 5
- FWCI
- 6.14
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Learning Technologies (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Education › Open Education and E-Learning
- Keywords
- expert ratings, consensus aggregation, matrix factorization, student performance data, community of learners
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.