Incorporating Training, Self-monitoring and AI-Assistance to Improve Peer Feedback Quality
Open accessAbstract
Peer review has been recognised as a beneficial approach that promotes higher-order learning and provides students with fast and detailed feedback on their work. Still, there are some common concerns and criticisms associated with the use of peer review that limits its adoption. One of the main points of concern is that feedback provided by students may be ineffective and of low quality. Previous works supply three explanations for why students may fail to provide effective feedback: They lack (1) the ability to provide high-quality feedback, (2) the agency to monitor their work or (3) the incentive to invest the required time and effort as they think the quality of the reviews are not reviewed. To help mitigate these shortcomings, this paper presents a complementary peer review approach that integrates training, self-monitoring and AI quality-control assistance to improve peer feedback quality. In particular, informed by higher education research, we built a set of training materials and a self-monitoring checklist for students to consider while writing their reviews. Also, informed by work from natural language processing, we developed quality control functions that automatically assess feedback submitted and prompt students to improve, if necessary. A between-subjects field experiment with 374 participants was conducted to investigate the approach's efficacy. Findings suggest that offering training, self-monitoring, and quality control functionalities to students assigned to the complementary peer review approach resulted in longer feedback that was perceived as more helpful than those who utilised the regular peer review interface. However, this complementary approach does not seem to affect students judgement (leniency or harshness) or confidence in grading. Directions are suggested to further evaluate and refine peer review systems.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
More commonly, systems display suggestions in a separated interface.
“More commonly, systems display suggestions in a separated interface [29,43,75,89].”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [29]
Section 3.2 describes training material accessed through a button and a checklist placed above the comment box; the Automatic Quality Control section also describes prompts shown to students. These passages support the existence of distinct interface elements for guidance, but they do not establish that systems more commonly display suggestions in a separated interface. (LLM, conf=0.78)
Experimental — accuracy not yet measuredDhillon et al.'s research demonstrated that while next-paragraph suggestions improved writing quality, longer AI text completions decreased satisfaction by undermining writers' independence.
“Similarly, Dhillon et al. 's [29] research demonstrated that while next-paragraph suggestions improved writing quality, longer AI text completions decreased satisfaction by undermining writers' independence.”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [29]
The AI-assistance and RQ2 sections discuss improving student peer-review feedback, not next-paragraph suggestions or longer AI text completions. Searches for Dhillon, text completions, writer independence, and satisfaction found no passage addressing the attributed finding. Page numbers were not available in our copy. (LLM, conf=0.78)
Experimental — accuracy not yet measuredStudies found that suggestion length is inversely associated with perceived ownership of the text.
“Other studies have examined the impact of suggestion length or quantity of suggestions on user experience and acceptance [14,29,36], finding that suggestion length is inversely associated with perceived ownership of the text.”
From Co-Writing with AI, on Human Terms: Aligning Research with User Demands Across the Writing Process (2025) · cites this paper as [29]
Section 4.2 reports the length of students’ feedback comments, and the “Specific” section reports how often those comments included improvement suggestions. Neither addresses whether the length of suggestions is inversely associated with perceived ownership of text. Searches for perceived ownership, suggestion length, and acceptance found no passage establishing that relationship. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 68 · Cited by 31
Research works printed but not linked: 82
- Solmaz Abdi, Hassan Khosravi, and Shazia Sadiq. 2020. Modelling learners in crowdsourcing educational systems. In International Conference on Artificial Intelligence in Education. Springer, 3-9.
- Solmaz Abdi, Hassan Khosravi, Shazia Sadiq, and Ali Darvishi. 2021. Open Learner Models for Multi-Activity Educational Systems. In International Conference on Artificial Intelligence in Education. Springer, 11-17.
- Solmaz Abdi, Hassan Khosravi, Shazia Sadiq, and Gianluca Demartini. 2021. Evaluating the Quality of Learning Resources: A Learnersourcing Approach. IEEE Transactions on Learning Technologies 14, 1 (2021), 81-92. https://doi.org/ 10.1109/TLT.2021.3058644 · DOI · Paper page
- Peter C Austin. 2011. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate behavioral research 46, 3 (2011), 399-424.
- Martine Baars, Lisette Wijnia, Anique de Bruin, and Fred Paas. 2020. The relation between student's effort and monitoring judgments during learning: a meta- analysis. Educational Psychology Review (2020), 1-24.
- Simon P Bates, Ross K Galloway, Jonathan Riise, and Danny Homer. 2014. As- sessing the quality of a student-generated question repository. Physical Review Special Topics-Physics Education Research 10, 2 (2014), 020105.
- David Boud, Ruth Cohen, and Jane Sampson. 2014. Peer learning in higher education: Learning from and with each other. Routledge.
- Susan Bull. 2020. There Are Open Learner Models About! IEEE Transactions on Learning Technologies (2020).
- Julia Cambre, Scott Klemmer, and Chinmay Kulkarni. 2018. Juxtapeer: Compara- tive peer review yields higher quality feedback and promotes deeper reflection. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1-13.
- David Carless. 2009. Trust, distrust and their impact on assessment reform. Assessment & Evaluation in Higher Education 34, 1 (2009), 79-89.
- David Carless, Kennedy Kam Ho Chan, Jessica To, Margaret Lo, and Elizabeth Barrett. 2018. Developing students' capacities for evaluative judgement through analysing exemplars. In Developing evaluative judgement in Higher Education. Routledge, 108-116.
- Anderson Pinheiro Cavalcanti, Arthur Diego, Rafael Ferreira Mello, Katerina Mangaroska, André Nascimento, Fred Freitas, and Dragan Gašević. 2020. How good is my feedback? a content analysis of written feedback. In Proceedings of the tenth international conference on learning analytics & knowledge. 428-437.
- Yea-huey Chang et al. 2016. Two decades of research in L2 peer review. Journal of Writing Research 8, 1 (2016).
- Ali Darvishi, Hassan Khosravi, and Shazia Sadiq. 2020. Utilising Learnersourc- ing to Inform Design Loop Adaptivity. In European Conference on Technology Enhanced Learning. Springer, 332-346.
- Ali Darvishi, Hassan Khosravi, and Shazia Sadiq. 2021. Employing peer review to evaluate the quality of student generated content at scale: A trust propagation approach. In Proceedings of the Eighth ACM Conference on Learning@ Scale. 139- 150.
- Ali Darvishi, Hassan Khosravi, Shazia Sadiq, and Barbara Weber. 2021. Neu- rophysiological Measurements in Higher Education: A Systematic Literature Review. International Journal of Artificial Intelligence in Education (2021), 1-41.
- Luca De Alfaro and Michael Shavlovsky. 2014. CrowdGrader: A tool for crowd- sourcing the evaluation of homework assignments. In Proceedings of the 45th ACM technical symposium on Computer science education. 415-420.
- Anne Pinot de Moira, Claire Massey, Jo-Anne Baird, and Marie Morrissy. 2002. Marking consistency over time. Research in Education 67, 1 (2002), 79-87.
- Paul Denny, Andrew Luxton-Reilly, and Beth Simon. 2009. Quality of student contributed questions using PeerWise. In Proceedings of the Eleventh Australasian Conference on Computing Education-Volume 95. 55-63.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). · arXiv
- Aslihan Erman Aslanoglu, Ismail Karakaya, and Mehmet Sata. 2020. Evaluation of University Students' Rating Behaviors in Self and Peer Rating Process via Many Facet Rasch Model. Eurasian Journal of Educational Research 89 (2020), 25-46.
- Kyle W Galloway and Simon Burns. 2015. Doing it for themselves: students creating a high quality peer-learning environment. Chemistry Education Research and Practice 16, 1 (2015), 82-92.
- Elena L Glassman, Aaron Lin, Carrie J Cai, and Robert C Miller. 2016. Learn- ersourcing personalized hints. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. 1626-1636.
- George Gyamfi, Barbara Hanna, and Hassan Khosravi. 2021. Supporting peer evaluation of student-generated content: a study of three approaches. Assessment & Evaluation in Higher Education 0, 0 (2021), 1-19. https://doi.org/10.1080/ 02602938.2021.2006140 · DOI
- George Gyamfi, Barbara E Hanna, and Hassan Khosravi. 2021. The effects of rubrics on evaluative judgement: a randomised controlled experiment. Assessment & Evaluation in Higher Education 47, 1 (2021), 126-143.
- John Hattie and Helen Timperley. 2007. The power of feedback. Review of educational research 77, 1 (2007), 81-112.
- Michael Henderson, Michael Phillips, Tracii Ryan, David Boud, Phillip Dawson, Elizabeth Molloy, and Paige Mahoney. 2019. Conditions that enable effective feedback. Higher Education Research & Development 38, 7 (2019), 1401-1416.
- Ken Hyland. 2019. Second language writing. Cambridge university press.
- Lasse X Jensen, Margaret Bearman, and David Boud. 2021. Understanding feed- back in online learning-A critical review and metaphor analysis. Computers & Education (2021), 104271.
- Qinjin Jia, Jialin Cui, Yunkai Xiao, Chengyuan Liu, Parvez Rashid, and Edward F Gehringer. 2021. ALL-IN-ONE: Multi-Task Learning BERT models for Evaluating Peer Assessments. arXiv preprint arXiv:2110.03895 (2021). · arXiv
- David A Joyner. 2017. Scaling expert feedback: Two case studies. In Proceedings of the Fourth (2017) ACM Conference on Learning@ Scale. 71-80.
- Douglas F Kauffman, Ruomeng Zhao, and Ya-Shu Yang. 2011. Effects of online note taking formats and self-monitoring prompts on learning from online text: Using technology to enhance self-regulated learning. Contemporary Educational Psychology 36, 4 (2011), 313-322.
- Hassan Khosravi, Kendra Cooper, and Kirsty Kitto. 2017. RiPLE: Recommendation in Peer-Learning Environments Based on Knowledge Gaps and Interests. JEDM| Journal of Educational Data Mining 9, 1 (2017), 42-67.
- Hassan Khosravi, Gianluca Demartini, Shazia Sadiq, and Dragan Gasevic. 2021. Charting the design and analytics agenda of learnersourcing systems. In LAK21: 11th International Learning Analytics and Knowledge Conference. 32-42.
- Hassan Khosravi, George Gyamfi, Barbara E. Hanna, Jason Lodge, and Solmaz Abdi. 2021. Bridging the Gap Between Theory and Empirical Research in Evalua- tive Judgment. Journal of Learning Analytics 8, 3 (2021), 117-132.
- Hassan Khosravi, Kirsty Kitto, and Williams Joseph. 2019. RiPPLE: A Crowd- sourced Adaptive Platform for Recommendation of Learning Activities. Journal of Learning Analytics 6, 3 (2019), 91-105.
- Hassan Khosravi, Shazia Sadiq, and Dragan Gasevic. 2020. Development and adoption of an adaptive learning system: Reflections and lessons learned. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. 58-64.
- Juho Kim. 2015. Learnersourcing: improving learning with collective learner activity. Ph. D. Dissertation. Massachusetts Institute of Technology.
- Juho Kim, Phu Tran Nguyen, Sarah Weir, Philip J Guo, Robert C Miller, and Krzysztof Z Gajos. 2014. Crowdsourcing step-by-step information extraction to enhance existing how-to videos. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 4017-4026.
- Vitomir Kovanović, Srećko Joksimović, Zak Waters, Dragan Gašević, Kirsty Kitto, Marek Hatala, and George Siemens. 2016. Towards automated content analysis of discussion transcripts: A cognitive presence case. In Proceedings of the sixth international conference on learning analytics & knowledge. 15-24.
- Markus Krause, Tom Garncarz, JiaoJiao Song, Elizabeth M Gerber, Brian P Bailey, and Steven P Dow. 2017. Critique style guide: Improving crowdsourced design feedback with a natural language model. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 4627-4639.
- Chinmay E Kulkarni, Michael S Bernstein, and Scott R Klemmer. 2015. Peer- Studio: rapid peer feedback emphasizes revision and improves performance. In Proceedings of the second (2015) ACM conference on learning@ scale. 75-84.
- Hatim Lahza, Hassan Khosravi, Gianluca Demartini, and Dragan Gasevic. 2022. Effects of Technological Interventions for Self-regulation: A Control Experi- ment in Learnersourcing. In LAK22: 12th International Learning Analytics and Knowledge Conference. 542-548.
- Daniël Lakens. 2013. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in psychology 4 (2013), 863.
- Hongli Li, Yao Xiong, Charles Vincent Hunter, Xiuyan Guo, and Rurik Tywoniw. 2020. Does peer assessment promote student learning? A meta-analysis. Assess- ment & Evaluation in Higher Education 45, 2 (2020), 193-211.
- Hsien-Chin Liou and Zhong-Yan Peng. 2009. Training effects on computer- mediated peer review. System 37, 3 (2009), 514-525.
- Ngar-Fun Liu and David Carless. 2006. Peer feedback: the learning element of peer assessment. Teaching in Higher education 11, 3 (2006), 279-290.
- Mario Manso-Vázquez and Martin Llamas-Nistal. 2015. A monitoring system to ease self-regulated learning processes. IEEE Revista Iberoamericana de Tecnologias del Aprendizaje 10, 2 (2015), 52-59.
- Sumie Matsuno. 2021. The Effects of Informing the Quality of Students' Previous Peer Assessment. In JACET International Convention Selected Papers, Vol. 1. The Japan Association of College English Teachers, 174.
- Amy McMillan, Pol Solanelles, and Bryan Rogers. 2021. Bias in student evalua- tions: Are my peers out to get me? Studies in Educational Evaluation 70 (2021), 101032.
- Rafael Molina-Carmona, Rosana Satorre-Cuerda, PATRICIA Compañ-Rosique, and Faraón Llorens-Largo. 2018. Metrics for estimating validity, reliability and bias in peer assessment. International Journal of Engineering Education 34, 3 (2018).
- T. Moon. 1996. The expectation-maximization algorithm. IEEE Signal Processing Magazine 13 (1996), 47-60.
- Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). 588- 593.
- Sapna Negi, Kartik Asooja, Shubham Mehrotra, and Paul Buitelaar. 2016. A study of suggestions in opinionated texts and their automatic detection. In Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics. 170-178.
- Melissa M Nelson and Christian D Schunn. 2009. The nature of feedback: How different types of peer feedback affect writing performance. Instructional science 37, 4 (2009), 375-401.
- Ikenna Osakwe, Guanliang Chen, Alex Whitelock-Wainwright, Dragan Gaševic, Anderson Pinheiro Cavalcanti, and Rafael Ferreira Mello. 2021. Towards auto- mated content analysis of feedback: A multi-language study. In Proceedings of the 14th International Conference on Educational Data Mining.
- Ellis B Page and Nancy S Petersen. 1995. The computer moves into essay grading: Updating the ancient test. Phi delta kappan 76, 7 (1995), 561.
- Jennifer Palisse, Deborah Martina King, and Mark MacLean. 2021. Comparative judgement and the hierarchy of students' choice criteria. International Journal of Mathematical Education in Science and Technology (2021), 1-21.
- Ernesto Panadero, Jesús Alonso Tapia, and Juan Antonio Huertas. 2012. Rubrics and self-assessment scripts effects on self-regulation, learning and self-efficacy in secondary education. Learning and individual differences 22, 6 (2012), 806-813.
- Dwayne E Paré and Steve Joordens. 2008. Peering into large lectures: examining peer and expert mark agreement using peerScholar, an online peer assessment tool. Journal of Computer Assisted Learning 24, 6 (2008), 526-540.
- Helen Purchase and John Hamer. 2018. Peer-review in practice: eight years of Aropä. Assessment & Evaluation in Higher Education 43, 7 (2018), 1146-1165.
- Lakshmi Ramachandran, Edward F Gehringer, and Ravi K Yadav. 2017. Automated assessment of the quality of peer reviews using natural language processing techniques. International Journal of Artificial Intelligence in Education 27, 3 (2017), 534-581.
- Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019). · arXiv
- Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Dis- tilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 (2019). · arXiv
- Bernhard Schmitz and Franziska Perels. 2011. Self-monitoring of self-regulation during math homework behaviour using standardized diaries. Metacognition and Learning 6, 3 (2011), 255-273.
- A Shibani. 2019. Augmenting Pedagogic Writing Practice with Contextualizable Learning Analytics. Ph. D. Dissertation.
- Victor Shnayder, Arpit Agarwal, Rafael Frongillo, and David C Parkes. 2016. Informed truthfulness in multi-task peer prediction. In Proceedings of the 2016 ACM Conference on Economics and Computation. 179-196.
- Victor Shnayder and David C Parkes. 2016. Practical peer prediction for peer assessment. In Fourth AAAI Conference on Human Computation and Crowdsourc- ing.
- Nancy Sommers. 1982. Responding to student writing. College composition and communication 33, 2 (1982), 148-156.
- Sean Tackett, Mark Raymond, Rishi Desai, Steven A Haist, Amy Morales, Shiv Gaglani, and Stephen G Clyman. 2018. Crowdsourcing for assessment items to support adaptive learning. Medical teacher 40, 8 (2018), 838-841.
- Timothy Teo. 2011. Technology acceptance in education. Springer Science & Business Media.
- Keith Topping. 2003. Self and peer assessment in school and university: Reliability, validity and utility. In Optimising new modes of assessment: In search of qualities and standards. Springer, 55-87.
- Jason L Walsh, Benjamin HL Harris, Paul Denny, and Phil Smith. 2018. Formative student-authored question bank: perceptions, question quality and association with summative performance. Postgraduate medical journal 94, 1108 (2018), 97-103.
- Wanyuan Wang, Bo An, and Yichuan Jiang. 2020. Optimal Spot-Checking for Improving the Evaluation Quality of Crowdsourcing: Application to Peer Grading Systems. IEEE Transactions on Computational Social Systems 7, 4 (2020), 940-955.
- David Kofoed Wind, Rasmus Malthe Jørgensen, and Simon Lind Hansen. 2018. Peer Feedback with Peergrade. In ICEL 2018 13th International Conference on e-Learning. Academic Conferences and publishing limited, 184.
- Philip H Winne. 2006. How software technologies can improve research on learning and bolster school reform. Educational psychologist 41, 1 (2006), 5-17.
- James R Wright, Chris Thornton, and Kevin Leyton-Brown. 2015. Mechanical TA: Partially automated high-stakes peer grading. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education. 96-101.
- Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016). · arXiv
- Wenting Xiong, D Litmaan, and Christian Schunn. 2012. Natural language processing techniques for researching and improving peer feedback. Journal of Writing Research 4, 2 (2012), 155-176.
- David Scott Yeager, Valerie Purdie-Vaughns, Julio Garcia, Nancy Apfel, Patti Brzustoski, Allison Master, William T Hessert, Matthew E Williams, and Geof- frey L Cohen. 2014. Breaking the cycle of mistrust: Wise interventions to provide critical feedback across the racial divide. Journal of Experimental Psychology: General 143, 2 (2014), 804.
- Qiyun Zhu and David Carless. 2018. Dialogue within peer feedback processes: Clarification and negotiation of meaning. Higher Education Research & Develop- ment 37, 4 (2018), 883-897.
- Barry J Zimmerman and Andrew S Paulsen. 1995. Self-monitoring during colle- giate studying: An invaluable tool for academic self-regulation. New directions for teaching and learning 1995, 63 (1995), 13-27.
- Authors
- Darvishi, A., Khosravi, H., Abdi, S., Sadiq, S., & Gašević, D.
- Affiliations
- The University of Queensland (AU)
- Monash University (AU)
- Venue
- Proceedings of the Ninth ACM Conference on Learning @ Scale
- Publisher
- ACM
- Published
- 2022-06
- Pages
- 35-47
- OpenAlex
- W4281687378
- Cited by
- 31
- References
- 68
- Citations by year
- 2026: 4 · 2025: 7 · 2024: 11 · 2023: 5 · 2022: 4
- FWCI
- 53.18
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Learning @ Scale (L@S) (publishedVersion, cc-by)
- Monash University Research Portal (Monash University) (submittedVersion, other-oa)
- HAL (submittedVersion)
- Funders
- The Australian Research Council's Industrial Transformation Training Centre for Information Resilience (CIRES) (IC200100022)
- Topics
- Social Sciences › Education › Student Assessment and Feedback
- Social Sciences › Psychology › Developmental and Educational Psychology › Innovative Teaching and Learning Methods
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Keywords
- peer review, higher education learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review