Juxtapeer: Comparative Peer Review Yields Higher Quality Feedback and Promotes Deeper Reflection
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Peer review asks novices to take on an evaluator's role, yet novices often lack the perspective to accurately assess the quality of others' work. To help learners give feedback on their peers' work through an expert lens, we present the Juxtapeer peer review system for structured comparisons. We build on theories of learning through contrasting cases, and contribute the first systematic evaluation of comparative peer review. In a controlled experiment, 476 consenting learners across four courses submitted 1,297 submissions, 4,102 reviews, and 846 self assessments. Learners assigned to compare submissions wrote reviews and self-reflections that were longer and received higher ratings from experts than those who evaluated submissions one at a time. A second study found that a ranking of submissions derived from learners' comparisons correlates well with staff ranking. These results demonstrate that comparing algorithmically-curated pairs of submissions helps learners write better feedback.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Formative assessment and the design of instructional systems
Sadler, D. R. (1989) · Instructional Science · doi:10.1007/bf00117714 · Cited by 4,285 · cited 1×
“However, these approaches fail to address an underlying problem: novices fail to give high quality feedback because they cannot differentiate good work from bad as well as instructors can, or even identify which aspects matter [41].”
BackgroundTo give is better than to receive: The benefits of peer review to the reviewer's own writing
Lundstrom, K., & Baker, W. (2009) · Journal of Second Language Writing · doi:10.1016/j.jslw.2008.06.002 · Cited by 843 · cited 1×
“At its best, peer review provides substantial benefit to both learners and instructors: reviewers gain experience providing constructive feedback on open-ended work [6,32], reviewees receive comments based on the diverse perspectives of multiple peers [10,48], and instructors reduce their feedback and grading burden d…”
BackgroundRecognition and retrieval processes in free recall.
Anderson, J. R., & Bower, G. H. (1972) · Psychological Review · doi:10.1037/h0033773 · Cited by 820 · cited 1×
“Just as “fortune cookies” leverage the idea that recognition is more reliable than recall [2], Juxtapeer’s guide helps students recognize four patterns for good feedback; feedback should be specific (comment on details rather than surface-level features), peer-oriented (provide constructive feedback to the peer, not j…”
BackgroundPeer and self assessment in massive online classes
Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2013) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2505057 · Cited by 390
Self-citationParallel prototyping leads to better design results, more divergence, and increased self-efficacy
Dow, S. P., Glassco, A., Kass, J., Schwarz, M., Schwartz, D. L., & Klemmer, S. R. (2010) · ACM Transactions on Computer-Human Interaction · doi:10.1145/1879831.1879836 · Cited by 367 · cited 1×
“Contrasting cases help develop nuanced understanding Across domains as diverse as algebra [39], design [12,49], and management [47], learners benefit from articulating the similarities and differences between examples [1].”
BackgroundSelf-citationGetting the right design and the design right
Tohidi, M., Buxton, W., Baecker, R., & Sellen, A. (2006) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1124772.1124960 · Cited by 211 · cited 1×
“Contrasting cases help develop nuanced understanding Across domains as diverse as algebra [39], design [12,49], and management [47], learners benefit from articulating the similarities and differences between examples [1].”
BackgroundPeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance
Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. R. (2015) · Proceedings of the Second (2015) ACM Conference on Learning @ Scale · doi:10.1145/2724660.2724670 · Cited by 192
Self-citationCalibrated peer review/sup TM/ and assessing learning outcomes
Carlson, P. A., & Berry, F. C. (2004) · 33rd Annual Frontiers in Education, 2003. FIE 2003. · doi:10.1109/fie.2003.1264740 · Cited by 43
Cited by
Wu, T., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517582 · Cited by 483
Critique Me: Exploring How Creators Publicly Request Feedback in an Online Critique Community
Cheng, R., Zeng, Z., Liu, M., & Dow, S. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415232 · Cited by 33
“Peer feedback systems for classrooms and MOOCs have focused on scaffolding feedback providers with rubrics [40] and prompts [53], reusing feedback from experts [46], and having peers compare multiple submissions [6].”
BackgroundIncorporating Training, Self-monitoring and AI-Assistance to Improve Peer Feedback Quality
Darvishi, A., Khosravi, H., Abdi, S., Sadiq, S., & Gašević, D. (2022) · Proceedings of the Ninth ACM Conference on Learning @ Scale · doi:10.1145/3491140.3528265 · Cited by 31
Epigraphics: Message-Driven Infographics Authoring
Zhou, T., Huang, J., & Chan, G. Y.-Y. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642172 · Cited by 17
“Gemini [32] is one such recommendation system that allows users to write declarative grammar to produce animated transitions between related statistical graphs.”
BackgroundLahza, H. F., Demartini, G., Noroozi, O., Gašević, D., Sadiq, S., & Khosravi, H. (2025) · Computers & Education · doi:10.1016/j.compedu.2025.105260 · Cited by 16
Metadata
- Authors
- Cambre, J., Klemmer, S., & Kulkarni, C.
- Venue
- Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2018-04-21
- Pages
- 1-13
- OpenAlex
- W2795596428
- Cited by
- 56
- References
- 8
- FWCI
- 40.27
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.