Calibrated peer review/sup TM/ and assessing learning outcomes
Abstract
The need for more focused and less-labor intensive assessment practices has brought new challenges, both for institutions and for individual educators. We elaborate on Calibrated Peer Review™ (CPR™) - an end- to-end computer-mediated learning environment that seamlessly integrates writing as a vehicle for critical thinking into a technical or content course. Developed as a tool to help incorporate writing into the teaching of science, CPR™ moves well beyond the scope of most web-delivered educational software. We draw from experiences using CPR™ in two courses offered at Rose -Hulman Institute of Technology, RH131 (Rhetoric and Composition) and ECE 360 (Principles of Engineering Design). We focus on four questions. First, what is CPR™ ? Second, how does CPR™ improve student learning? Third, how can CPR™ measure learning outcomes appropriate for an ABET-style assessment? And fourth, does the system serve as a de facto electronic portfolio in engineering education?
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vygotsky, L. S. (1964) · Bulletin of the Orton Society · doi:10.1007/bf02928399 · Cited by 20,704 · see citing works
Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956) · Cited by 9,342 · see citing works
(1956) · Medical Entomology and Zoology · Cited by 4,596 · see citing works
Emig, J. A. (2020) · Routledge eBooks · doi:10.4324/9781003059219-10 · Cited by 854 · see citing works
Emig, J. A. (1977) · College Composition and Communication · doi:10.2307/356095 · Cited by 755 · see citing works
In the Age of the Smart Machine: The Future of Work and Power
Zuboff, S. (1989) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2073307 · Cited by 410 · see citing works
In the age of the smart machine: The future of work and power
(1989) · Long Range Planning · doi:10.1016/0024-6301(89)90205-7 · Cited by 298 · see citing works
Looking Back as We Look Forward: Historicizing Writing Assessment
Yancey, K. B. (1999) · College Composition and Communication · doi:10.2307/358862 · Cited by 193 · see citing works
Four Effective Writing Strategies for Engineering Classes
Sharp, J. E., Olds, B., Miller, R. L., & Dyrud, M. A. (1999) · Journal of Engineering Education · doi:10.1002/j.2168-9830.1999.tb00411.x · Cited by 85 · see citing works
Robinson, R. (2001) · The American Biology Teacher · doi:10.1662/0002-7685(2001)063[0474:cpr]2.0.co;2 · Cited by 69 · see citing works
Rethinking the evaluation of writing in engineering courses
Swarts, J., & Odell, L. (2002) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2001.963905 · Cited by 24 · see citing works
Undergraduate Learning Portfolios for Institutional Assessment
Heinricher, A. C., Miller, J. E., Schachterle, L. E., Kildahl, N. K., Bluemel, V., & Crawford, V. M. (2002) · Journal of Engineering Education · doi:10.1002/j.2168-9830.2002.tb00699.x · Cited by 21 · see citing works
Engineering Thinking: Using Benjamin Bloom and William Perry to Design Assignments
Irish, R. K. (1999) · Language and Learning Across the Disciplines · doi:10.37514/lld-j.1999.3.2.06 · Cited by 21 · see citing works
Improving technical writing via web-based peer review of final reports
Eschenbach, E. A. (2002) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2001.963724 · Cited by 16 · see citing works
The role(s) of assessment in technical communication: A review of the literature
Allen, J. (1993) · Technical Communication Quarterly · doi:10.1080/10572259309364548 · Cited by 15 · see citing works
HYPERTEXT TOOLS FOR KNOWLEDGE WORKERS
Carlson, P. A., & Slaven, M. (1992) · Information Systems Management · doi:10.1080/10580539208906866 · Cited by 10 · see citing works
Self-citation
Flagged first, then by citations
1 of the 17 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Artificial intelligence for education: Knowledge and its assessment in AI-enabled learning ecologies
Cope, B., Kalantzis, M., & Searsmith, D. (2021) · Educational Philosophy and Theory · doi:10.1080/00131857.2020.1728732 · Cited by 518 · see citing works
Peer and self assessment in massive online classes
Kulkarni, C., Wei, K. P., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. R. (2013) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2505057 · Cited by 390 · see citing works
PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance
Kulkarni, C. E., Bernstein, M. S., & Klemmer, S. R. (2015) · Proceedings of the Second (2015) ACM Conference on Learning @ Scale · doi:10.1145/2724660.2724670 · Cited by 192 · see citing works
Understanding user perception toward artificial intelligence (AI) enabled e-learning
Kashive, N., Powale, L., & Kashive, K. (2020) · The International Journal of Information and Learning Technology · doi:10.1108/ijilt-05-2020-0090 · Cited by 181 · see citing works
Technology-Mediated Writing Assessments: Principles and Processes
Cope, B., Kalantzis, M., McCarthey, S. J., Vojak, C., & Kline, S. M. (2011) · Computers & composition/Computers and composition · doi:10.1016/j.compcom.2011.04.007 · Cited by 133 · see citing works
Peer and Self Assessment in Massive Online Classes
Kulkarni, C. E., Koh, P. W., Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., & Klemmer, S. (2014) · Understanding innovation · doi:10.1007/978-3-319-06823-7_9 · Cited by 104 · see citing works
Scaling short-answer grading by combining peer assessment with algorithmic scoring
Kulkarni, C. E., Socher, R., Bernstein, M. S., & Klemmer, S. (2014) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2556325.2566238 · Cited by 95 · see citing works
Chiang, C., Kasunic, A., & Savage, S. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274306 · Cited by 57 · see citing works
Cambre, J., Klemmer, S., & Kulkarni, C. E. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173868 · Cited by 56 · see citing works
A Fuzzy Group Decision Making Model for Ordinal Peer Assessment
Capuano, N., Loia, V., & Orciuoli, F. (2016) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2016.2565476 · Cited by 41 · see citing works
Peer assessment in the algorithms course
Chinn, D. D. (2005) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1067445.1067468 · Cited by 34 · see citing works
The student view on online peer reviews
Bauer, C., Figl, K., Derntl, M., Beran, P. P., & Kabicher, S. (2009) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/1562877.1562892 · Cited by 31 · see citing works
Towards Adaptive Peer Assessment for MOOCs
Capuano, N., & Caballé, S. (2015) · International Conference on P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC) · doi:10.1109/3pgcic.2015.7 · Cited by 24 · see citing works
Peer and Self-Assessment for First-Year Students as a Tool to Improve Learning
van Hattum-Janssen, N., & Lourenço, J. M. (2008) · Journal of Professional Issues in Engineering Education and Practice · doi:10.1061/(asce)1052-3928(2008)134:4(346 · Cited by 24 · see citing works
Calibrated Peer Review Essays Increase Student Confidence in Assessing Their Own Writing.
Likkel, L. (2012) · The journal of college science teaching · Cited by 21 · see citing works
Calibrated Peer Review for Interpreting Linear Regression Parameters: Results from a Graduate Course
Enders, F. B., Jenkins, S. M., & Hoverman, V. R. (2010) · Journal of Statistics Education · doi:10.1080/10691898.2010.11889489 · Cited by 20 · see citing works
FOPA-MC: fuzzy multi-criteria group decision making for peer assessment
Capuano, N., Caballé, S., Percannella, G., & Ritrovato, P. (2020) · Soft Computing · doi:10.1007/s00500-020-05155-5 · Cited by 15 · see citing works
The student view on online peer reviews
Bauer, C., Figl, K., Derntl, M., Beran, P. P., & Kabicher, S. (2009) · ACM SIGCSE Bulletin · doi:10.1145/1595496.1562892 · Cited by 14 · see citing works
Calibrated Peer Review Assignments for the Earth Sciences
Rudd, J. A., Wang, V. Z., Spencer‐Cervato, C., & Ridky, R. W. (2009) · Journal of Geoscience Education · doi:10.5408/1.3559673 · Cited by 13 · see citing works
Did I mention it's anonymous? The triumphs and pitfalls of online peer review
Rourke, A. J., Mendelssohn, J., Coleman, K. S., & Allen, B. (2008) · Cited by 11 · see citing works
Essig, R. R., Troy, C. D., Jesiek, B. K., Boyd, J., & Buswell, N. T. (2020) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--20032 · Cited by 10 · see citing works
Volz, T. M., & Saterbak, A. (2009) · Across the Disciplines · doi:10.37514/atd-j.2009.6.2.14 · Cited by 9 · see citing works
Designing Scalable and Sustainable Peer Interactions Online
Kulkarni, C. E., Kotturi, Y., Bernstein, M. S., & Klemmer, S. (2016) · Understanding innovation · doi:10.1007/978-3-319-40382-3_14 · Cited by 7 · see citing works
Application of Fuzzy Ordinal Peer Assessment in Formative Evaluation
Capuano, N., & Orciuoli, F. (2017) · International Conference on P2P, Parallel, Grid, Cloud and Internet Computing · doi:10.1007/978-3-319-69835-9_52 · Cited by 5 · see citing works
Improving Writing Quality of Capstone Reports
Berry, F., Phillips, M., Condron, J., & Sanger, P. A. (2021) · IEEE Transactions on Education · doi:10.1109/te.2021.3059739 · Cited by 5 · see citing works
Self-citationCollaborative Calibrated Peer Assessment in Massive Open Online Courses
Boudria, A., Lafifi, Y., & Bordjiba, Y. (2017) · International Journal of Distance Education Technologies · doi:10.4018/ijdet.2018010105 · Cited by 4 · see citing works
A Systematic Analysis of Peer Assessment in the MOOC Era and Future Perspectives
Wahid, U., Schroeder, U., & Chatti, M. A. (2016) · RWTH Publications (RWTH Aachen) · Cited by 3 · see citing works
Multi-criteria Fuzzy Ordinal Peer Assessment for MOOCs
Capuano, N., & Caballé, S. (2018) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-3-319-98557-2_34 · Cited by 3 · see citing works
Designing A Process For Department Curricular Reform
Froyd, J. E., Layne, J., Yurttas, L., & Ford, D. N. (2020) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--650 · Cited by 3 · see citing works
Practical Interpretation of Student Evaluations for Starting Professors
Schilling, W. W., Estell, J. K., & Berry, F. (2020) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--18907 · Cited by 2 · see citing works
Self-citationRobledo, D. C. (2011) · OakTrust (Texas A&M University Libraries) · Cited by 2 · see citing works
CPRTM: Adopting An Out-Of-Discipline Innovation
Strong, K. E. (2011) · College Teaching Methods & Styles Journal (CTMS) · doi:10.19030/ctms.v4i1.5051 · Cited by 1 · see citing works
Incorporating peer review of course term project in structural analysis course
Barroso, L. R., & Morgan, J. (2010) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2010.5673245 · Cited by 1 · see citing works
Teaching freshmen engineering communication
Larsen, B. B., Lundheim, L., Ekman, T., & Tybell, T. (2016) · European Workshop on Microelectronics Education (EWME) · doi:10.1109/ewme.2016.7496464 · Cited by 1 · see citing works
A Graph Analysis Method to Improve Peer Grading Accuracy for Blended Teaching Courses
Du, X., Wang, X., & Ma, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3136352 · Cited by 1 · see citing works
Pessoa, A. S. M., Coelho, L., & Fernandes, R. (2015) · Advances in higher education and professional development book series · doi:10.4018/978-1-4666-7304-5.ch005 · Cited by 2 · see citing works
Collaborative Calibrated Peer Assessment in Massive Open Online Courses
Boudria, A., Lafifi, Y., & Bordjiba, Y. (2019) · IGI Global eBooks · doi:10.4018/978-1-7998-0420-8.ch065 · Cited by 1 · see citing works
Froyd, J. E., Layne, J. G., Yurttas, L., Ford, D. N., & Ford, D. M. (2006)
AC2011-533: INCORPORATINGTECHNICALPEERREVIEWOFCIVIL ENGINEERING STUDENT PROJECTS
Barroso, L. R., & Morgan, J. (2011)
Rourke, A. J., Mendelssohn, J., Coleman, K. S., & Allen, B. (2008) · ASCILITE Publications · doi:10.14742/apubs.2008.2384 · Cited by 11 · see citing works
Learning to Assess Software Quality: A Metric-Guided Peer Code Review Tool for Large-Scale Software
Tillou, A., Mayor, P. O., Rixner, S., & Schreib, R. (2026) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3803400.3809374
William, W., & Ho, J. (2015) · Huria Journal of the Open University of Tanzania
- Authors
- Carlson, P. A., & Berry, F. C.
- Affiliations
- Rose–Hulman Institute of Technology (US)
- University of Colorado Boulder (US)
- ORCID
- Berry, F.C.
- Venue
- 33rd Annual Frontiers in Education, 2003. FIE 2003.
- Publisher
- IEEE
- Published
- 2004-07-08
- Volume
- 2
- Pages
- F3E_1-F3E_6
- ISSN
- 0190-5848 (ISSN-L)
- OpenAlex
- W2099691798
- Cited by
- 43
- References
- 17
- Citations by year
- 2026: 1 · 2021: 3 · 2020: 6 · 2019: 1 · 2018: 3 · 2017: 2 · 2016: 4 · 2015: 4 · 2014: 2 · 2013: 1 · 2012: 1
- FWCI
- 4.98
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Frontiers in Education Conference (FIE) (publishedVersion)
- Rose-Hulman Scholar (Rose–Hulman Institute of Technology) (submittedVersion)
- Topics
- Social Sciences › Education › Student Assessment and Feedback
- Physical Sciences › Engineering › Media Technology › Engineering Education and Curriculum Development
- Physical Sciences › Computer Science › Information Systems › Educational Technology and Assessment
- Keywords
- Calibrated Peer Review, peer review, learning outcomes assessment, ABET accreditation, engineering education, writing across the curriculum, critical thinking, computer-mediated learning environments, e-portfolio, engineering design courses, rhetoric and composition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
