Automated Assessment of the Quality of Peer Reviews using Natural Language Processing Techniques
Open accessThe DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Abstract
A review is textual feedback provided by a reviewer to the author of a submitted version. Peer reviews are used in academic publishing and in education to assess student work. While reviews are important to e-commerce sites like Amazon and e-bay, which use them to assess the quality of products and services, our work focuses on academic reviewing. We seek to help reviewers improve the quality of their reviews. One way to measure review quality is through metareview or review of reviews. We develop an automated metareview software that provides rapid feedback to reviewers on their assessment of authors’ submissions. To measure review quality, we employ metrics such as: review content type, review relevance, review’s coverage of a submission, review tone, review volume and review plagiarism (from the submission or from other reviews). We use natural language processing and machine-learning techniques to calculate these metrics. We summarize results from experiments to evaluate our review quality metrics: review content, relevance and coverage, and a study to analyze user perceptions of importance and usefulness of these metrics. Our approaches were evaluated on data from Expertiza and the Scaffolded Writing and Rewriting in the Discipline (SWoRD) project, which are two collaborative web-based learning applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The nature of feedback: How different types of peer feedback affect writing performance
Nelson, M. M., & Schunn, C. D. (2009) · Instructional Science · doi:10.1007/s11251-008-9053-x · Cited by 602 · cited 12×
“In order to combine the two datasets for our evaluation, SWoRD reviews that were coded as summary or praise were treated as summative reviews, and reviews coded as explicit problems are treated as problem detection reviews, while those coded as explicit solutions are treated as advisory reviews.”
BackgroundMethodInfluentialDevelopment of the Review Quality Instrument (RQI) for Assessing Peer Reviews of Manuscripts
van Rooyen, S., Black, N., & Godlee, F. (1999) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(99)00047-5 · Cited by 183 · cited 2×
“Rooyen et al. (1999) use the RQI to check whether the reviewer discusses the following: (1) importance of the research question, (2) originality, (3) strengths and weaknesses, (4) presentation and interpretation of results.”
BackgroundMethodReviewing the reviewers: A study of author perception on peer reviews in computer science
Kühne, C., Böhm, K., & Yue, J. Z. (2010) · Proceedings of the 6th International ICST Conference on Collaborative Computing: Networking, Applications, Worksharing · doi:10.4108/icst.collaboratecom.2010.33 · Cited by 8
Cited by
Artificial intelligence for education: Knowledge and its assessment in AI-enabled learning ecologies
Cope, B., Kalantzis, M., & Searsmith, D. (2021) · Educational Philosophy and Theory · doi:10.1080/00131857.2020.1728732 · Cited by 518
“Still another has been to design “ MetaScholar, ” in order to offer initial machine feedback on human textual feedback so it can be improved before submitting to a peer (Ramachandran, Gehringer, & Yadav, 2017).”
BackgroundTan, S. C., Lee, A. V. Y., & Lee, M. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100097 · Cited by 164
Ferreira‐Mello, R., André, M., Pinheiro, A., Costa, E., & Romero, C. (2019) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1332 · Cited by 193
Incorporating AI and learning analytics to build trustworthy peer assessment systems
Darvishi, A., Khosravi, H., Sadiq, S., & Gašević, D. (2022) · British Journal of Educational Technology · doi:10.1111/bjet.13233 · Cited by 120
Wu, Y., & Schunn, C. D. (2023) · Contemporary Educational Psychology · doi:10.1016/j.cedpsych.2023.102160 · Cited by 78
Metadata
- Authors
- Ramachandran, L., Gehringer, E. F., & Yadav, R. K.
- Venue
- International Journal of Artificial Intelligence in Education
- Published
- 2017-01-11
- Volume
- 27
- Issue
- 3
- Pages
- 534-581
- ISSN
- 1560-4292 (ISSN-L)
- 1560-4306
- OpenAlex
- W2573443938
- Cited by
- 61
- References
- 3
- FWCI
- 2.30
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Automated Assessment of the Quality of Peer Reviews using Natural Language Processing Techniques. https://app.cochat.ai/papers/10.1007/s40593-016-0132-x
- The paper
- Ramachandran, L., Gehringer, E. F., & Yadav, R. K. (2017). Automated Assessment of the Quality of Peer Reviews using Natural Language Processing Techniques. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-016-0132-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.