PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance
Open accessAbstract
Rapid feedback is a core component of mastery learning, but feedback on open-ended work requires days or weeks in most classes today. This paper introduces PeerStudio, an assessment platform that leverages the large number of students' peers in online classes to enable rapid feedback on in-progress work. Students submit their draft, give rubric-based feedback on two peers' drafts, and then receive peer feedback. Students can integrate the feedback and repeat this process as often as they desire. In MOOC deployments, the median student received feedback in just twenty minutes. Rapid feedback on in-progress work improves course outcomes: in a controlled experiment, students' final grades improved when feedback was delivered quickly, but not if delayed by 24 hours. More than 3,600 students have used PeerStudio in eight classes, both massive and in-person. This research demonstrates how large classes can leverage their scale to encourage mastery through rapid feedback and revision.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Self-monitoring has been incorporated within the peer review process to enhance the quality of peer feedback.
“Notably, self-monitoring has also been incorporated within the peer review process to enhance the quality of peer feedback. For instance, Kulkarni, Bernstein, and Klemmer (2015) proposed the use of scaffold comments within PeerStudio, allowing students to re-review their peer feedback before its final submission.”
From Impact of AI assistance on student agency (2024) · cites this paper as Kulkarni, Bernstein, and Klemmer (2015)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hattie, J., & Timperley, H. (2007) · Review of Educational Research · doi:10.3102/003465430298487 · Cited by 12,905 · see citing works
The role of deliberate practice in the acquisition of expert performance
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993) · Psychological Review · doi:10.1037/0033-295x.100.3.363 · Cited by 9,564 · cited 1× · see citing works
“To most effectively develop mastery, students must repeatedly revise based on immediate, focused feedback [12].”
BackgroundKluger, A. N., & DeNisi, A. (1996) · Psychological Bulletin · doi:10.1037/0033-2909.119.2.254 · Cited by 5,882 · see citing works
Peer and self assessment in massive online classes
Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2013) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2505057 · Cited by 390 · cited 3× · see citing works
“Prior work shows peer-based critique is effective both for in-person [8,29] and online classes [25], and can provide students accurate numeric grades and comments [13,25].”
BackgroundSelf-citationCalibrated peer review/sup TM/ and assessing learning outcomes
Carlson, P. A., & Berry, F. C. (2004) · 33rd Annual Frontiers in Education, 2003. FIE 2003. · doi:10.1109/fie.2003.1264740 · Cited by 43 · see citing works
Flagged first, then by citations
Research works printed but not linked: 23
- Anderson, S. and Rodin, J. Is Bad News Always Bad?: Cue and Feedback Effects on Intrinsic Motivation. Journal of Applied Social Psychology 19, 6 (1989), 449-467.
- Andrade, H.G. The Effects of Instructional Rubrics on Learning to Write. Current Issues in Education 4, 4 (2001).
- Andrade, H.G. Teaching with rubrics: The good, the bad, and the ugly. College Teaching 53, 1 (2005), 27-31.
- André, P., Bernstein, M., and Luther, K. Who gives a tweet? Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work -CSCW '12, ACM Press (2012), 471.
- Balcazar, F.E., Hopkins, B.L., and Suarez, Y. A critical, objective review of performance feedback. Journal of Organizational Behavior Management 7, 2 (1986).
- Buxton, B. Sketching user experiences: getting the design right and the right design. Morgan Kaufmann, 2007.
- Carlson, P.A. and Berry, F.C. Calibrated Peer Review and assessing learning outcomes. Frontiers in Education Conference, (2003).
- Dawes, R.M. The robust beauty of improper linear models in decision making. American psychologist 34, 7 (1979), 571.
- Dow, S., Fortuna, J., Schwartz, D., Altringer, B., and Klemmer, S. Prototyping dynamics: sharing multiple designs improves exploration, group rapport, and results. Proceedings of the 2011 annual conference on Human factors in computing systems, (2011), 2807-2816.
- Dow, S.P., Heddleston, K., and Klemmer, S.R. The efficacy of prototyping under time constraints. Proceeding of the ACM conference on Creativity and cognition, ACM Press (2009), 165.
- Falchikov, N. and Goldfinch, J. Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of educational research 70, 3 (2000), 287-322.
- Fast, E., Lee, C., Aiken, A., Bernstein, M.S., Koller, D., and Smith, E. Crowd-scale interactive formal reasoning and analytics. Proceedings of the 26th annual ACM symposium on User interface software and technology, (2013).
- Gray, W.D. and Boehm-Davis, D.A. Milliseconds matter: an introduction to microstrategies and to their use in describing and predicting interactive behavior. Journal of experimental psychology. Applied 6, 4 (2000), 322-35.
- Guskey, T.R. Closing Achievement Gaps: Revisiting Benjamin S. Bloom's "Learning for Mastery." Journal of Advanced Academics 19, 1 (2007), 8-31.
- Kim, S.-M., Pantel, P., Chklovski, T., and Pennacchiotti, M. Automatically assessing review helpfulness. Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics (2006), 423-430.
- Kittur, A., Chi, E.H., and Suh, B. Crowdsourcing user studies with Mechanical Turk. Proc. of CHI, ACM Press (2008), 453.
- Krosnick, J.A. Survey research. Annual review of psychology 50, 1 (1999), 537-567.
- Kulik, J.A. and Kulik, C.-L.C. Timing of Feedback and Verbal Learning. Review of Educational Research 58, 1 (1987), 79- 97.
- Kulkarni, C., Socher, R., Bernstein, M.S., and Klemmer, S.R. Scaling Short-answer Grading by Combining Peer Assessment with Algorithmic Scoring. ACM Conf on Learning@Scale, (2014).
- Latham, G.P. and Locke, E.A. Self-regulation through goal setting. Organizational Behavior and Human Decision Processes 50, 2 (1991), 212-247.
- Marsh, R.L., Landau, J.D., and Hicks, J.L. How examples may (and may not) constrain creativity. Memory & cognition 24, 5 (1996), 669-80.
- Sommers, N. Responding to Student Writing. College Composition and Communication 33, 2 (1982), 148-156.
- Tinapple, D., Olson, L., and Sadauskas, J. CritViz: Web- Based Software Supporting Peer Critique in Large Creative Classrooms. Bulletin of the IEEE Technical Committee on Learning Technology 15, 1 (2013), 29.
Cited by
Sorted by citations
Impact of AI assistance on student agency
Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & Siemens, G. (2024) · Computers & Education · doi:10.1016/j.compedu.2023.104967 · Cited by 548 · see citing works
- Authors
- Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. R.
- Affiliations
- Stanford University (US)
- UC San Digeo, La Jolla, CA, USA
- Venue
- Proceedings of the Second (2015) ACM Conference on Learning @ Scale
- Publisher
- ACM
- Published
- 2015-03-14
- Pages
- 75-84
- OpenAlex
- W2038924341
- Cited by
- 192
- References
- 5
- Citations by year
- 2026: 9 · 2025: 13 · 2024: 15 · 2023: 13 · 2022: 14 · 2021: 24 · 2020: 20 · 2019: 15 · 2018: 24 · 2017: 27 · 2016: 17 · 2015: 1
- FWCI
- 145.76
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Conference on Learning @ Scale (L@S) (publishedVersion)
- Funders
- National Science Foundation (1351131, 1444865)
- Topics
- Social Sciences › Education › Student Assessment and Feedback
- Physical Sciences › Computer Science › Computer Science Applications › Online Learning and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- peer feedback, rapid feedback, learning mastery, open-ended assessment, MOOCs, large classes, course outcomes, revision, online learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
