CoChat
Sign Up
Cited by 120Student Assessment and FeedbackOnline Learning and AnalyticsArtificial Intelligence in Education

Incorporating AI and learning analytics to build trustworthy peer assessment systems

Open access
Darvishi, Ali, Khosravi, Hassan, Sadiq, Shazia, Gašević, Dragan·2022·British Journal of Educational Technology
DOI 10.1111/bjet.13233
Open this paper in CoChat — free accountOpen PDF

Abstract

Peer assessment has been recognised as a sustainable and scalable assessment method that promotes higher‐order learning and provides students with fast and detailed feedback on their work. Despite these benefits, some common concerns and criticisms are associated with the use of peer assessments (eg, scarcity of high‐quality feedback from peer student‐assessors and lack of accuracy in assigning a grade to the assessee) that raise questions about their trustworthiness. Consequently, many instructors and educational institutions have been anxious about incorporating peer assessment into their teaching. This paper aims to contribute to the growing literature on how AI and learning analytics may be incorporated to address some of the common concerns associated with peer assessment systems, which in turn can increase their trustworthiness and adoption. In particular, we present and evaluate our AI‐assisted and analytical approaches that aim to (1) offer guidelines and assistance to student‐assessors during individual reviews to provide better feedback, (2) integrate probabilistic and text analysis inference models to improve the accuracy of the assigned grades, (3) develop feedback on reviews strategies that enable peer assessors to review the work of each other, and (4) employ a spot‐checking mechanism to assist instructors in optimally overseeing the peer assessment process. Practitioner notes: What is already known about this topic Engaging students in peer assessment has been demonstrated to have various benefits. However, there are some common concerns associated with employing peer assessment that raise questions about their trustworthiness as an assessment item. What this paper adds Methods and processes on how AI and learning analytics may be incorporated to address some of the common concerns associated with peer assessment systems, which in turn, can increase their trustworthiness and adoption. Implications for practice Presentation of a systematic approach for development, deployment and evaluation of AI and analytics approaches in peer assessment systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1111/bjet.13233

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 58

  1. S Abdi, S., Khosravi, H., & Sadiq, S. (2020). Modelling learners in crowdsourcing educational systems. In International Conference on Artificial Intelligence in Education (pp. 3-9). Springer.
  2. Abdi, S., Khosravi, H., Sadiq, S., & Darvishi, A. (2021). Open learner models for multi-activity educational sys- tems. Artificial Intelligence in Education, 11-17. https://doi.org/10.1007/978-3-030-78270 -2_2 · DOI
  3. Ahmad, N., & Bull, S. (2008). Do students trust their open learner models? In International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems (pp. 255-258). Springer.
  4. Baars, M., Wijnia, L., de Bruin, A., & Paas, F. (2020). The relation between student's effort and monitoring judg- ments during learning: A meta-analysis. Educational Psychology Review, 32, 1-24.
  5. Bodily, R., Kay, J., Aleven, V., Jivet, I., Davis, D., Xhakaj, F., & Verbert, K. (2018). Open learner models and learn- ing analytics dashboards: a systematic review. In Proceedings of the 8th international conference on learning analytics and knowledge (pp. 41-50). Association for Computing Machinery.
  6. Carless, D. (2020). From teacher transmission of information to student feedback literacy: Activating the learner role in feedback processes. Active Learning in Higher Education, 1469787420945845. https://doi. org/10.1177/14697 87420 945845 · DOI
  7. Darvishi, A., Khosravi, H., & Sadiq, S. (2020). Utilising learner sourcing to inform design loop adaptivity. In European Conference on Technology Enhanced Learning (pp. 332-346). Springer.
  8. Darvishi, A., Khosravi, H., & Sadiq, S. (2021). Employing peer review to evaluate the quality of student generated content at scale: A trust propagation approach. In Proceedings of the Eighth ACM Conference on Learning@ Scale (pp. 139-150). Association for Computing Machinery.
  9. Darvishi, A., Khosravi, H., Sadiq, S., & Weber, B. (2021). Neurophysiological Measurements in Higher Education: A Systematic Literature Review. International Journal of Artificial Intelligence in Education. https://doi. org/10.1007/s4059 3-021-00256 -0 · DOI
  10. Darvishi, A., Khosravi, H., Abdi, S., Sadiq, S., & Gašević, D. (2022). Incorporating training, self-monitoring and AI-assistance to improve peer feedback quality. In Proceedings of the Ninth ACM Conference on Learning @ Scale (L@S '22), June 1-3, 2022, New York City, NY, USA. ACM. https://doi.org/10.1145/34911 40.3528265 14678535, 2022, 4, · DOI · Paper page
  11. De Alfaro, L., & Shavlovsky, M. (2014). CrowdGrader: A tool for crowdsourcing the evaluation of homework assignments. In Proceedings of the 45th ACM technical symposium on Computer science education (pp. 415-420). Association for Computing Machinery.
  12. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. · arXiv
  13. El Maarry, K., Güntzer, U., & Balke, W.-T. (2015). A majority of wrongs doesn't make it right-On crowdsourcing quality for skewed domain tasks. In International Conference on Web Information Systems Engineering (pp. 293-308). Springer, Cham.
  14. Gardner, J., Brooks, C., & Baker, R. (2019). Evaluating the fairness of predictive student models through slic- ing analysis. In Proceedings of the 9th international conference on learning analytics & knowledge (pp. 225-234). Association for Computing Machinery.
  15. Gyamfi, G., Hanna, B. E., & Khosravi, H. (2021). The effects of rubrics on evaluative judgement: A ran- domised controlled experiment. Assessment & Evaluation in Higher Education, 47(1), 126-143. https://doi. org/10.1080/02602 938.2021.1887081 · DOI
  16. Hadwin, A., Järvelä, S., & Miller, M. (2017). Self-regulation, co-regulation, and shared regulation in collaborative learning environments. In Handbook of self-regulation of learning and performance (pp. 83-106). Routledge.
  17. Han, Y., Wu, W., Yan, Y., & Zhang, L. (2020). Human-machine hybrid peer grading in SPOCs. IEEE Access, 8, 220922-220934.
  18. Hassan, T. (2019). Trust and trustworthiness in social recommender systems. In Companion Proceedings of The 2019 World Wide Web Conference (pp. 529-532). Association for Computing Machinery.
  19. Huisman, B., Admiraal, W., Pilli, O., van de Ven, M., & Saab, N. (2018). Peer assessment in MOOCs: The re- lationship between peer reviewers' ability and authors' essay performance. British Journal of Educational Technology, 49, 101-110.
  20. Joyner, D. A. (2017). Scaling expert feedback: Two case studies. In Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale (pp. 71-80). Association for Computing Machinery.
  21. Kao, G. Y.-M. (2013). Enhancing the quality of peer review by reducing student "free riding": Peer assessment with positive interdependence. British Journal of Educational Technology, 44, 112-124.
  22. Khosravi, H., Kitto, K., & Joseph, W. (2019). Ripple: A crowdsourced adaptive platform for recommendation of learning activities. Journal of Learning Analytics, 6, 91-105.
  23. Khosravi, H., Demartini, G., Sadiq, S., & Gasevic, D. (2021). Charting the design and analytics agenda of learn- ersourcing systems. In LAK21: 11th International Learning Analytics and Knowledge Conference (LAK21) (pp. 32-42). Association for Computing Machinery. https://doi.org/10.1145/34481 39.3448143 · DOI
  24. Khosravi, H., Gyamfi, G., Hanna, B. E., Lodge, J., & Abdi, S. (2021). Bridging the gap between theory and empir- ical research in evaluative judgment. Journal of Learning Analytics, 8(3), 117-132. https://doi.org/10.18608/ jla.2021.7206 · DOI
  25. Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. R. (2015). PeerStudio: rapid peer feedback emphasizes revision and improves performance. In Proceedings of the second (2015) ACM conference on learning@ scale (pp. 75-84). Association for Computing Machinery.
  26. Lahza, H., Khosravi, H., Demartini, G., & Gasevic, D. (2022). Effects of technological interventions for self- regulation: A control experiment in learnersourcing. In LAK22: 12th International Learning Analytics and Knowledge Conference. Association for Computing Machinery. https://doi.org/10.1145/35068 60.3506911 · DOI
  27. Lee, W., Huang, C. H., Chang, C. W., Wu, M. K. D., Chuang, K. T., Yang, P. A. and Hsieh, C. C. (2018) Effective quality assurance for data labels through crowdsourcing and domain expert collaboration. In 21st International Conference on Extending Database Technology, EDBT 2018 (pp. 646-649). OpenProceedings.org.
  28. Levy, H., & Robinson, M. (2006). Stochastic dominance: Investment decision making under uncertainty (Vol. 34). Springer.
  29. Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor or assessee: How student learning improves by giving and receiving peer feedback. British Journal of Educational Technology, 41, 525-536.
  30. Li, Y., Jin, X., Hu, Q., Jiang, Q., Zhao, W., & Oubibi, M. (2021). An empirical study on the influence of co-regulation on deep learning under crowdsourcing knowledge construction. In 2021 IEEE 24th International Conference on Computer Supported Cooperative Work in Design (CSCWD) (pp. 526-530). IEEE.
  31. Lockey, S., Gillespie, N. and Curtis, C. (2020) Trust in artificial intelligence: Australian insights. The University of Queensland and KPMG. https://doi.org/10.14264/ b32f129 · DOI
  32. Manso-Vázquez, M., & Llamas-Nistal, M. (2015). A monitoring system to ease self-regulated learning processes. IEEE Revista Iberoamericana de Tecnologias del Aprendizaje, 10, 52-59.
  33. Martinez-Maldonado, R., Gaševic, D., Echeverria, V., Fernandez Nieto, G., Swiecki, Z., & Buckingham Shum, S. (2021). What do you mean by collaboration analytics? A conceptual model. Journal of Learning Analytics, 8, 126-153.
  34. Matcha, W., Gašević, D., & Pardo, A. (2019). A systematic review of empirical studies on learning analytics dashboards: A self-regulated learning perspective. IEEE Transactions on Learning Technologies, 13(2), 226-245.
  35. Moon, T. (1996). The expectation-maximization algorithm. IEEE Signal Processing Magazine, 13, 47-60.
  36. Napoles, C., Sakaguchi, K., Post, M., & Tetreault, J. (2015). Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (pp. 588-593). Association for Computational Linguistics (ACL).
  37. Negi, S., Asooja, K., Mehrotra, S., & Buitelaar, P. (2016). A study of suggestions in opinionated texts and their automatic detection. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics (pp. 170-178). Association for Computational Linguistics.
  38. Nejad, A. M., & Mahfoodh, O. H. A. (2019). Assessment of oral presentations: Effectiveness of self-, peer-, and teacher assessments. International Journal of Instruction, 12, 615-632.
  39. Paré, D. E., & Joordens, S. (2008). Peering into large lectures: Examining peer and expert mark agreement using peerscholar, an online peer assessment tool. Journal of Computer Assisted Learning, 24, 526-540.
  40. Polisda, Y. (2017). Peer review: A strategy to improve students' academic essay writings. English Franca: Academic Journal of English Language and Education, 1, 45-60.
  41. Purchase, H., & Hamer, J. (2018). Peer-review in practice: Eight years of Aropä. Assessment & Evaluation in Higher Education, 43, 1146-1165.
  42. Ramachandran, L., Gehringer, E. F., & Yadav, R. K. (2017). Automated assessment of the quality of peer reviews using natural language processing techniques. International Journal of Artificial Intelligence in Education, 27, 534-581.
  43. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 3982-3992). Association for Computational Linguistics.
  44. Robinson, S. L. (1996). Trust and breach of the psychological contract. Administrative Science Quarterly, 41, 574-599. 14678535, 2022, 4, · DOI
  45. Shin, D. (2020). User perceptions of algorithmic decisions in the personalized AI system: Perceptual evaluation of fairness, accountability, transparency, and explainability. Journal of Broadcasting & Electronic Media, 64, 541-565.
  46. Shnayder, V., Agarwal, A., Frongillo, R., & Parkes, D. C. (2016). Informed truthfulness in multi-task peer prediction. In Proceedings of the 2016 ACM Conference on Economics and Computation (pp. 179-196). Association for Computing Machinery.
  47. Shoham, N., & Pitman, A. (2021). Open versus blind peer review: Is anonymity better than transparency? BJPsych Advances, 27, 247-254.
  48. Tao, D., Cheng, J., Yu, Z., Yue, K., & Wang, L. (2018). Domain-weighted majority voting for crowdsourcing. IEEE Transactions on Neural Networks and Learning Systems, 30, 163-174.
  49. Topping, K. (2003). Self and peer assessment in school and university: Reliability, validity and utility. In Optimising new modes of assessment: In search of qualities and standards (pp. 55-87). Springer, Dordrecht.
  50. Urena, R., Kou, G., Dong, Y., Chiclana, F., & Herrera-Viedma, E. (2019). A review on trust propagation and opinion dynamics in social networks and group decision making frameworks. Information Sciences, 478, 461-475.
  51. Wang, W., An, B. and Jiang, Y. (2018) Optimal spot-checking for improving evaluation accuracy of peer grading systems. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 32, No. 1). AAAI Press.
  52. Wang, W., An, B., & Jiang, Y. (2020). Optimal spot-checking for improving the evaluation quality of crowdsourcing: Application to peer grading systems. IEEE Transactions on Computational Social Systems, 7, 940-955.
  53. Wind, D. K., Jørgensen, R. M., & Hansen, S. L. (2018). Peer feedback with peergrade. In ICEL 2018 13th International Conference on e-Learning (p. 184). Academic Conferences and Publishing Limited.
  54. Wright, J. R., Thornton, C., & Leyton-Brown, K. (2015). Mechanical TA: Partially automated high-stakes peer grading. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (pp. 96-101). Association for Computing Machinery.
  55. Xiong, W., & Litman, D. (2011). Automatically predicting peer-review helpfulness. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (pp. 502-507). Association for Computational Linguistics.
  56. Yang, M., Tai, M., & Lim, C. P. (2016). The role of e-portfolios in supporting productive learning. British Journal of Educational Technology, 47, 1276-1286.
  57. Yang, T.-Y., Baker, R. S., Studer, C., Heffernan, N., & Lan, A. S. (2019). Active learning for student affect detec- tion. In Proceedings of the 12th International Conference on Educational Data Mining, EDM 2019, Montréal, Canada, July 2-5, 2019. International Educational Data Mining Society (IEDMS) 2019 (pp. 208-217). Université du Québec; Polytechnique Montréal.
  58. Yeager, D. S., Purdie-Vaughns, V., Garcia, J., Apfel, N., Brzustoski, P., Master, A., Hessert, W. T., Williams, M. E., & Cohen, G. L. (2014). Breaking the cycle of mistrust: Wise interventions to provide critical feedback across the racial divide. Journal of Experimental Psychology: General, 143, 804-824.

Cited by

Sorted by citations

Authors
Darvishi, A., Khosravi, H., Sadiq, S., & Gašević, D.
Affiliations
  • The University of Queensland (AU)
  • Monash University (AU)
Corresponding author
  • Darvishi, Ali
  • Khosravi, Hassan
ORCID
Venue
British Journal of Educational Technology
Publisher
Wiley
Published
2022-05-18
Volume
53
Issue
4
Pages
844-875
ISSN
  • 0007-1013 (ISSN-L)
  • 1467-8535
DOI
10.1111/bjet.13233
OpenAlex
W4280598345
Cited by
120
References
18
Citations by year
2026: 34 · 2025: 33 · 2024: 32 · 2023: 19 · 2022: 2
FWCI
46.39
Type
journal article
Language
en
Open access
Licence
Locations
  • British Journal of Educational Technology (publishedVersion, cc-by-nc)
  • Monash University Research Portal (Monash University) (submittedVersion, other-oa)
  • HAL (submittedVersion)
Funders
University of Queensland
Topics
  • Social Sciences › Education › Student Assessment and Feedback
  • Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords
peer assessment, artificial intelligence in education, learning analytics, student feedback, peer review, spot-checking, higher-order learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review