MOPRD: A multidisciplinary open peer review dataset
Open accessAbstract
Open peer review is a growing trend in academic publications. Public access to peer review data can benefit both the academic and publishing communities. It also serves as a great support to studies on review comment generation and further to the realization of automated scholarly paper review. However, most of the existing peer review datasets do not provide data that cover the whole peer review process. Apart from this, their data are not diversified enough as the data are mainly collected from the field of computer science. These two drawbacks of the currently available peer review datasets need to be addressed to unlock more opportunities for related studies. In response, we construct MOPRD, a multidisciplinary open peer review dataset. This dataset consists of paper metadata, multiple version manuscripts, review comments, meta-reviews, author's rebuttal letters, and editorial decisions. Moreover, we propose a modular guided review comment generation method based on MOPRD. Experiments show that our method delivers better performance as indicated by both automatic metrics and human evaluation. We also explore other potential applications of MOPRD, including meta-review generation, editorial decision prediction, author rebuttal generation, and scientometric analysis. MOPRD is a strong endorsement for further studies in peer review-related research and other applications.
Verified 2026-10-29 · 4 references not checked yet
AI snapshot
Written from the paper’s full text.
TL;DR
MOPRD is a multidisciplinary open peer review dataset built from PeerJ, containing 6,578 papers with metadata, multiple manuscript versions, review comments, meta-reviews, rebuttal letters, and editorial decisions. The authors also propose a modular guided review comment generation method and report gains over an extract-then-generate baseline on automatic and human metrics.
Key findings
- MOPRD contains 6,578 papers, 22,483 review comments, and 11,213 rebuttal letters; mean review rounds per manuscript is 2.7, mean reviewers per manuscript is 2.4, mean v1 manuscript word count is 5,434, mean v2 manuscript word count is 6,015, mean v1 review comment word count per reviewer is 636, mean v2 review comment word count per reviewer is 224, and mean meta-review word count is 129.
- The proposed modular guided method outperformed the extract-then-generate method of Yuan et al. (2022) on ROUGE-1 (39.01 vs. 35.59), ROUGE-2 (9.10 vs. 8.57), ROUGE-Lsum (37.53 vs. 33.99), and BARTScore (−4.15 vs. −4.19), but not on ROUGE-L (15.49 vs. 16.10).
- In human evaluation, the proposed method scored higher than the compared method on structure & logicality (3.45 vs. 2.85), specialty & accuracy (3.30 vs. 2.75), and usefulness & inspiration (3.40 vs. 2.65).
- Multi-discipline training outperformed single-discipline training on all reported metrics: ROUGE-1 30.96 vs. 29.97, ROUGE-2 7.98 vs. 7.52, ROUGE-L 14.25 vs. 14.07, ROUGE-Lsum 29.88 vs. 28.88, and BARTScore −4.23 vs. −4.25.
Methods
- Data were collected from PeerJ using special crawlers, covering paper metadata, initial submissions, revisions, review comments, meta-reviews, author rebuttal letters, and editorial decisions; MOPRD is provided as Native MOPRD and Processed MOPRD, with data collected on Aug 16, 2022.
- For review comment generation, PDFs were converted with GROBID, section titles were classified into summary, methods & results, and others using a SciBERT-based model (accuracy 0.986, weighted F1 0.986), and LED was fine-tuned for five modules: Basic Reporting, Experiments & Findings, Questions, Proposals, and Additional Comments; Questions and Proposals were auto-labeled using NLTK heuristics.
- Experiments used 10,998 manuscript-comment pairs from initial submissions, filtered to manuscripts ≥2,000 words and review comments ≥100 words, split 8:1:1; evaluation used ROUGE and BARTScore, plus human evaluation with 10 researchers scoring 20 generated review comments on a five-point scale.
Limitations
- The paper states that exhaustive collection of all open peer review data was not achieved, that PeerJ was selected as the data source, and that MOPRD is a long-term project intended to cover more journals and disciplines when resources allow.
- The paper notes concerns about multidisciplinary bias, reports that generated text may contain grammatical errors and inexplicable phrasing due to Top-P/Top-K sampling, and explains that models performed worse when generating for an unknown discipline with smaller training data.
- Evident from the text: the dataset is limited to PeerJ’s seven journals and the disciplines represented there, the data snapshot is from Aug 16, 2022, the human evaluation is small (10 researchers, 10 manuscripts, 20 review comments), and the auto-labeling of Questions and Proposals relies on heuristics such as question marks, keywords, and POS tagging.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
8 of 14 cited sources checked · 5 supported · 0 flagged · accuracy not yet measured
- Premisecitation§ 4.1 Method
The maximum input length of previous generation of deep learning generative models, such as BART, is only 1,024 tokens.
“the maximum input length of the previous generation of deep learning generative models, such as BART(Lewis et al., 2020), is only 1,024 tokens.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Lewis et al., 2020 source
The Architecture (Section 2.1) and Generation Tasks (Section 5.3) passages establish that BART is used for text generation, but neither gives a maximum input length. A search for “1024 tokens” and “maximum sequence length” found 1024 only as the model’s hidden size in Experimental Setup (Section 5.1), not as a token limit. Our copy therefore does not verify the claimed 1,024-token maximum. (LLM, conf=0.72)
Experimental — accuracy not yet measured
- Premisecitation§ 1 Introduction
Open peer review is proved to attract review comments of higher quality in terms of length, helpfulness, and good manners.
“Open peer review is proved to attract review comments of higher quality in terms of length, helpfulness, and good manners (Bornmann et al., 2012; Walsh et al., 2000).”
Quote in the paperNot a substantial claim1 of 2 sources checked- cites Bornmann et al., 2012 source
- cites Walsh et al., 2000 source
The Review quality section (p. 2) reports higher overall quality for signed reviews, including more constructive comments, and significantly greater courtesy. But the paper does not establish that the comments were longer: the longer duration reported on p. 2 is time spent completing a review, not comment length. The claim’s unqualified 'proved' also overstates a finding from this trial. (LLM, conf=0.88)
Experimental — accuracy not yet measured
- Findingown data§ 4.3.3 Comparison of generation methods over human evaluation
The proposed method has the highest score of 3.45 for structure & logicality.
“Compared to all other scores, our method has the highest score of 3.45 for structure & logicality.”
Quote in the paperSubstantial claim - Findingreasoning§ Proposals
The extract-then-generate method has the highest ROUGE-L score among all methods.
“The extract-then-generate method has the highest ROUGE-L score among all methods.”
Quote in the paperSubstantial claim - Findingreasoning§ Proposals
The modular non-guided method surpasses the segmentation-less method in most metrics.
“The modular non-guided method surpasses the segmentation-less method in most metrics.”
Quote in the paperSubstantial claim
- Findingown data§ Abstract
MOPRD is a multidisciplinary open peer review dataset.
“In response, we construct MOPRD, a multidisciplinary open peer review dataset.”
Quote in the paperSubstantial claim- Rests on an assumption: PeerJ is a large general academic publisher of up to seven journals covering a wide range of research disciplines.
- Findingown data§ 3.3 Properties
MOPRD contains 22,483 review comments.
“| Total review comments | 22,483 |”
Quote in the paperNot a substantial claim - Findingown data§ 3.3 Properties
MOPRD contains 11,213 rebuttal letters.
“| Total rebuttal letters | 11,213 |”
Quote in the paperNot a substantial claim - Findingreasoning§ 4.3 Experiments
The final dataset for the experiments consists of 10,998 records of manuscript-comment pairs.
“The final dataset for the experiments consists of 10,998 records of manuscript-comment pairs.”
Quote in the paperSubstantial claim - Findingown data§ Proposals
The proposed modular guided method outperforms all other methods over all metrics except ROUGE-L in comparison with the extract-then-generate method.
“As displayed in Table 3, our proposed method outperforms all other methods over all metrics except ROUGE-L in the comparison with the extractthen-generate method.”
Quote in the paperSubstantial claim - Findingreasoning§ Proposals
The segmentation-less method delivers greater performance than the extract-then-generate method over all other metrics.
“The segmentation-less method delivers greater performance than the extract-then-generate method over all other metrics.”
Quote in the paperNot a substantial claim - Findingown data§ Abstract
The proposed modular guided review comment generation method based on MOPRD delivers better performance as indicated by both automatic metrics and human evaluation.
“Experiments show that our method delivers better performance as indicated by both automatic metrics and human evaluation.”
Quote in the paperSubstantial claim - Findingreasoning§ 4.1 Method
With PeerJ as data source, review comments in MOPRD are already labeled as they are originally composed of four different segments.
“With PeerJ being the data source, the review comments in MOPRD are already being labeled as they are originally composed of four different segments.”
Quote in the paperSubstantial claim- Rests on an assumption: Review comments on PeerJ are well-structured with four segments: Basic Reporting, Experimental Design, Validity of the Findings, and Additional Comments.
- Findingown data§ 4.3.3 Comparison of generation methods over human evaluation
The proposed method surpasses the compared method over all human evaluation metrics.
“As shown in Table 4, our method surpasses the compared method over all metrics.”
Quote in the paperSubstantial claim - Findingown data§ 3.3 Properties
MOPRD contains 6,578 papers.
“In total, 6,578 papers are collected in MOPRD.”
Quote in the paperNot a substantial claim - Findingown data§ 4.3.3 Comparison of generation methods over human evaluation
The scores of the proposed method are all above 3 points.
“In addition, the scores of our method are all above 3 points.”
Quote in the paperNot a substantial claim - Findingown data§ 4.3.4 Single-discipline training data VS multi-discipline training data
MOPRD mainly consists of data from five disciplines: biology, chemistry, computer science, environment, and medicine.
“As displayed in Fig. 2, MOPRD mainly consists of data from five disciplines: biology, chemistry, computer science, environment, and medicine.”
Quote in the paperNot a substantial claim - Findingreasoning§ 4.3.4 Single-discipline training data VS multi-discipline training data
Review comments generated in this experiment show lower scores.
“Review comments generated in this experiment are found to show lower scores.”
Quote in the paperSubstantial claim - Findingnone given§ 6 Conclusion
MOPRD is the first large-scale open peer review dataset of multidisciplinary data.
“In this paper, we introduce the Multidisciplinary Open Peer Review Dataset (MOPRD), which is, to our best knowledge, the first large-scale open peer review dataset of multidisciplinary data.”
Quote in the paperSubstantial claim - Findingown data§ 6 Conclusion
In review comment generation for unknown disciplines, models trained with multi-discipline data work better than models trained with single-discipline data.
“Our experiments also demonstrate that in review comment generation for unknown disciplines, models trained with multi-discipline data work better than models trained with single-discipline data.”
Quote in the paperSubstantial claim - Findingnone given§ 6 Conclusion
The data cover the entire peer review process from paper submission to publication.
“The data cover the entire peer review process from paper submission to publication.”
Quote in the paperSubstantial claim - Findingown data§ 6 Conclusion
The modular guided method is superior to the compared methods as measured by ROUGE and BARTScore.
“This method is superior to the compared methods as measured by the automatic metrics of ROUGE and BARTScore.”
Quote in the paperSubstantial claim - Findingreasoning§ 4.3.4 Single-discipline training data VS multi-discipline training data
There is performance inconsistency between the models in this experiment and the models in the previous experiments.
“It is also noted that there is performance inconsistency between the models in this experiment and the models in the previous experiments, which can be seen in Table 3.”
Quote in the paperSubstantial claim - Findingown data§ 4.3.3 Comparison of generation methods over human evaluation
The rating for specialty & accuracy is relatively lower.
“In terms of specialty & accuracy, the rating is relatively lower.”
Quote in the paperNot a substantial claim - Findingown data§ 4.3.4 Single-discipline training data VS multi-discipline training data
The multi-discipline model delivers better performance in review comment generation than the single-discipline model by all metrics.
“As displayed in Table 5, it is indicated by all metrics that the multidiscipline model can deliver better performance in review comment generation than the single-discipline model.”
Quote in the paperSubstantial claim - Findingown data§ 5.1 Meta-review generation
MOPRD stands out with completeness of peer review data because initial submissions and all revisions are collected.
“Compared to those datasets, MOPRD stands out with its completeness of peer review data because the initial submissions and all the revisions of the papers are collected.”
Quote in the paperSubstantial claim - Findingnone given§ 6 Conclusion
The proposed method delivers the best performance in structure & logicality among the three aspects.
“The proposed method delivers the best performance in the aspect of structure & logicality among the three aspects.”
Quote in the paperNot a substantial claim - Premisecitation§ 4.2 Discussion
A simple end-to-end model can produce better results when trained with large-scale data, compared with a complex model trained with small-scale data.
“Additionally, Soltau et al. (2017) have proved that a simple end-to-end model can produce better results when trained with large-scale data, compared with a complex model trained with small-scale data.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Soltau et al. (2017) source
- Interpretationreasoning§ 4.2 Discussion
Papers other than PeerJ can use the modular guided method equally.
“Therefore, papers other than PeerJ can use our method equally.”
Quote in the paperSubstantial claim- Rests on an assumption: PeerJ review comments are composed of four segments: Basic Reporting, Experimental Design, Validity of the Findings, and Additional Comments.
- Rests on an assumption: Most papers have review comments in a structure similar to PeerJ's.
- Interpretationreasoning§ 4.2 Discussion
With LED and modular manuscript segmentation, long manuscripts can be processed effectively.
“With the use of LED and the approach of modular manuscript segmentation, long manuscripts can be processed effectively.”
Quote in the paperSubstantial claim - Interpretationown data§ Proposals
The effectiveness of the proposed method is validated.
“This validates the effectiveness of our proposed method.”
Quote in the paperSubstantial claim - Interpretationown data§ Proposals
Using full text rather than extracted summary as input is more effective in feature representation.
“This reveals that using full text rather than extracted summary as input is more effective in feature representation.”
Quote in the paperSubstantial claim - Premisereasoning§ 1 Introduction
A large proportion of presently available peer review datasets do not include papers across different disciplines.
“a large proportion of presently available datasets do not include papers across different disciplines.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4.2 Discussion
The modular guided method can be reused for most papers.
“Our modular guided method is developed based on the review comments of PeerJ but, nonetheless, it is an accommodating method that can be reused for most papers.”
Quote in the paperSubstantial claim- Rests on an assumption: PeerJ review comments are composed of four segments: Basic Reporting, Experimental Design, Validity of the Findings, and Additional Comments.
- Rests on an assumption: Most papers have review comments in a structure similar to PeerJ's.
- Premisereasoning§ 1 Introduction
The majority of publicly available open peer review datasets are datasets of computer science.
“The majority of the publicly available open peer review datasets are datasets of computer science.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ Proposals
Without the guided design, the modular design cannot make the most of its potential.
“However, without the guided design, the modular design can not make the most of it.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4.2 Discussion
With enough data and appropriate selection and balancing strategies, it is possible to learn enough features across disciplines.
“With enough data and appropriate selection and balancing strategies, it is consequently possible to learn enough features across disciplines in this interdisciplinary trend.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ 4.2 Discussion
The multidisciplinary bias of the method does not invalidate it.
“However, this situation does not invalidate our method.”
Quote in the paperNot a substantial claim- Rests on an assumption: Papers in different disciplines have various writing styles and focuses.
- Premisecitation§ 4.2 Discussion
Research methods from different disciplines continue to merge.
“Nowadays, interdisciplinary research is on the rise (Van Noorden, 2015), and research methods from different disciplines also continue to merge.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites Van Noorden, 2015 source
- Interpretationreasoning§ Proposals
The extract-then-generate method's high ROUGE-L score is due to its use of beam search with 4 beams, which tends to generate phrases with the four highest-probability words.
“This is not surprising as the extractthen-generate method uses beam search with the number of beams set to 4. This design tends to generate phrases with the four words of the highest probability, hence the high ROUGE-L score.”
Quote in the paperSubstantial claim - Interpretationcitation§ 4.2 Discussion
Large-scale datasets can be a performance booster for AI research.
“Therefore, large-scale datasets can be a performance booster for AI research.”
Quote in the paperSubstantial claim - Interpretationown data§ Proposals
The modular design is capable of better performance.
“This indicates that the modular design is capable of better performance.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 2 Related work
MOPRD is by far the largest multidisciplinary peer review dataset with complete peer review history.
“to our best knowledge, MOPRD is by far the largest multidisciplinary peer review dataset with complete peer review history.”
Quote in the paperNot a substantial claim - Premisereasoning§ 1 Introduction
Most existing peer review datasets do not provide data that cover the whole peer review process.
“most of the currently existing datasets do not contain data that cover the whole peer review process.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ Proposals
The modular design and the guided design need to work together to achieve the best results.
“The modular design and the guided design need to work together to achieve the best results.”
Quote in the paperSubstantial claim - Interpretationown data§ 4.3.4 Single-discipline training data VS multi-discipline training data
Models trained with multi-discipline data are more capable of transferring learned features in generating review comments for unknown disciplines.
“This proves that models trained with multidiscipline data are more capable of transferring the learned features in the generation of review comments for unknown disciplines.”
Quote in the paperSubstantial claim - Study design and conductnone given§ 4.3.4 Single-discipline training data VS multi-discipline training data
The biology data used for single-discipline training are not overlapped with those biology data used for multi-discipline training.
“The biology data used for single-discipline training are not overlapped with those biology data used for multi-discipline training.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4.3.3 Comparison of generation methods over human evaluation
ASPR is a realistic goal.
“This shows that ASPR is to be achieved as a realistic goal.”
Quote in the paperNot a substantial claim - Interpretationown data§ 6 Conclusion
MOPRD is more suitable for real-life applications than previous datasets.
“This is the reason why MOPRD is more suitable for real-life applications than the previous datasets.”
Quote in the paperNot a substantial claim - Interpretationown data§ 4.3.4 Single-discipline training data VS multi-discipline training data
MOPRD as a multi-discipline dataset can rectify the inadequacy of existing datasets to generate proper review comments for papers from multiple disciplines.
“But MOPRD as a multi-discipline dataset can rectify this inadequacy to generate proper review comments for papers from multiple disciplines.”
Quote in the paperSubstantial claim- Rests on an assumption: Most existing review comment datasets are comprised either entirely or mostly of data from the discipline of computer science only.
- Interpretationreasoning§ 4.3.4 Single-discipline training data VS multi-discipline training data
Existing review comment datasets are inadequate for review comment generation for papers from other disciplines.
“These datasets are inadequate for review comment generation for papers from other disciplines.”
Quote in the paperSubstantial claim- Rests on an assumption: Most existing review comment datasets are comprised either entirely or mostly of data from the discipline of computer science only.
- Interpretationown data§ 4.3.4 Single-discipline training data VS multi-discipline training data
Another reason for the lower scores is that the training data size is significantly smaller, halved compared to previous experiments.
“Another reason is that the size of training data for the models in this experiment is significantly smaller compared to that of the models in the previous experiments. The training data are halved in size in this experiment.”
Quote in the paperSubstantial claim - Interpretationreasoning§ 4.3.4 Single-discipline training data VS multi-discipline training data
One reason for the lower scores is that the models are applied to generate review comments for an unknown discipline in training.
“One reason is that the models in this experiment are applied to generate review comments for an unknown discipline in the training.”
Quote in the paperSubstantial claim
- Classified as probably not a substantial claim. 39
- May combine two claims. 19
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 4.1.2 Preparation for generationLED extends the input length limit to 16,384 tokens.
“which extends the input length limit to 16,384 tokens.”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Beltagy et al., 2020 source
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 1 IntroductionAs of December 2019, a total of 617 journals have taken up the practice of open peer review.
“As of December 2019, a total of 617 journals have taken up the practice of open peer review (Wolfram et al., 2020).”
Quote in the paperNot a substantial claim1 of 1 sources checked- cites Wolfram et al., 2020 source
Classified as probably not a substantial claim.
Premisecitation§ 4.1 MethodSpecial attention mechanisms such as Big Bird, LED, and LongT5 are designed to ease input limits for long-text input.
“special attention mechanisms are designed to ease the input limits for the processing of long-text input. Some examples include Big Bird (Zaheer et al., 2020), Longformer-Encoder-Decoder (LED) (Beltagy et al., 2020), and LongT5 (Guo et al., 2022).”
Quote in the paperSubstantial claim2 of 3 sources checkedClassified as probably not a substantial claim.
Premisecitation§ 4 Review comment generationReview comment generation is one of the key steps in achieving ASPR.
“Review comment generation is one of the key steps in achieving ASPR.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Lin et al., 2023 source
May combine two claims.
Findingown data§ 5.4 Scientometric analysisMOPRD has 20% more data from a wider variety of disciplines than the data used in Matsui et al. (2021).
“MOPRD has 20% more data from a wider variety of disciplines than the data used in this study.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites Matsui et al. (2021) source
The Data > PeerJ data section (page unavailable) identifies the disciplines included in Matsui et al.’s dataset and gives article counts, supporting the baseline for the comparison. It does not describe MOPRD, so it cannot establish that MOPRD has 20% more data or covers a wider variety of disciplines. (LLM, conf=0.80)
Experimental — accuracy not yet measured
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingreasoning§ 4.3.1 Sanity testA small portion of the output may exhibit grammatical errors and inexplicable phrasing due to the combined use of Top-P and Top-K sampling.
“Given the combined utilization of Top-P and Top-K sampling for content selection during generation, a small portion of the output may exhibit grammatical errors and inexplicable phrasing.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Findingreasoning§ 4.1 MethodThe paper designs a strategy of modular generation by segmenting input manuscripts based on the structure of review comments.
“we design the strategy of modular generation by segmenting the input manuscripts based on the structure of the review comments.”
Quote in the paperSubstantial claimMay combine two claims.
Findingreasoning§ ProposalsThe modular guided method surpasses both the segmentation-less method and the modular non-guided method over all metrics.
“Overall, the modular guided method surpasses both the segmentation-less method and the modular non-guided method over all metrics.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ 3.3 PropertiesMean v1 manuscript word count in MOPRD is 5,434.
“| Mean v1 manuscript word count | 5,434 |”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ 3.3 PropertiesMean v2 manuscript word count in MOPRD is 6,015.
“| Mean v2 manuscript word count | 6,015 |”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ 3.3 PropertiesMean review rounds per manuscript in MOPRD is 2.7.
“| Mean review rounds per manuscript | 2.7 |”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Findingreasoning§ 4.1 MethodAuto-labeling methods are proposed for automatic Questions and Proposals labeling.
“auto-labeling methods are proposed in this paper for automatic Questions and Proposals labeling.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Findingnone given§ 4.2 DiscussionMOPRD is the largest multidisciplinary open peer review dataset by far.
“MOPRD, as the largest multidisciplinary open peer review dataset by far, is hopeful to continually elicit more and better data from the research community.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown data§ 4.1.1 PreprocessingA SciBERT-based classification method achieves an accuracy of 0.986 and a weighted F1-score of 0.986 for segmenting manuscripts into three subsets.
“The trained model achieves an accuracy of 0.986 and a weighted F1-score of 0.986.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.
Findingown data§ 4.3.1 Sanity testThe generated review comment addresses the focal points of each target module and is pertinent to the content of the paper itself.
“As shown in Fig. 4, the generated review comment does address the focal points of each target module, while also being pertinent to the content of the paper itself.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Findingreasoning§ 4.1.2 Preparation for generationThe paper takes the pioneering step to tap into data of questions and proposals for review comment generation.
“in this study, we take the pioneering step to tap into the data of questions and proposals for review comment generation.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ 3.3 PropertiesMean v2 review comment word count per reviewer in MOPRD is 224.
“| Mean v2 review comment word count per reviewer | 224 |”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ 3.3 PropertiesMean meta-review word count in MOPRD is 129.
“| Mean meta-review word count | 129 |”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Findingnone given§ 4.1.3 GenerationIn the modular guided method, each model is guided to generate specific contents for the corresponding module.
“in our modular guided method, each model is guided to generate specific contents for the corresponding module.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
Findingnone given§ 4.3.1 Sanity testThe generated review comment shows no conventional rigidity associated with template-based generation.
“Moreover, there is no conventional rigidity, which is often associated with template-based generation, shown in the generated review comment.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ 3.3 PropertiesMean v1 review comment word count per reviewer in MOPRD is 636.
“| Mean v1 review comment word count per reviewer | 636 |”
Quote in the paperNot a substantial claimMay combine two claims.
Findingnone given§ AbstractMOPRD consists of paper metadata, multiple version manuscripts, review comments, meta-reviews, author's rebuttal letters, and editorial decisions.
“This dataset consists of paper metadata, multiple version manuscripts, review comments, meta-reviews, author's rebuttal letters, and editorial decisions.”
Quote in the paperSubstantial claim- Rests on an assumption: PeerJ peer review history includes initial submission, revisions, review comments, meta-reviews, author's rebuttal letters, and editorial decisions.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingnone given§ 4.1.2 Preparation for generationR_ques and R_propos are not native data of PeerJ; they are obtained by the authors through an auto-labeling method.
“r ques i and r propos i are not native data of PeerJ. They are obtained by the authors through an auto-labeling method.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ 3.2 ConstructionMOPRD is available in two versions: Native MOPRD and Processed MOPRD.
“MOPRD is available in two versions. One is the Native MOPRD that contains all the files crawled from the web pages in HTML format as well as the PDF, DOC, and DOCX files downloaded. It is provided as an accommodating version for users to process and convert the open peer review data as needed. The other version is the Processed MOPRD.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown data§ 3.3 PropertiesMean reviewer number per manuscript in MOPRD is 2.4.
“| Mean reviewer number per manuscript | 2.4 |”
Quote in the paperNot a substantial claimMay combine two claims.
Findingnone given§ 6 ConclusionThe method completely surpasses the compared method in all aspects of structure & logicality, specialty & accuracy, and usefulness & inspiration in human evaluation, with every aspect score above the mean.
“The method also completely surpasses the compared method in all aspects of structure & logicality, specialty & accuracy, and usefulness & inspiration in human evaluation, with every aspect score above the mean.”
Quote in the paperSubstantial claimMay combine two claims.
Findingnone given§ 4.3.4 Single-discipline training data VS multi-discipline training dataBiology is the largest discipline with the most data in MOPRD.
“Data from the discipline of biology are used for single-discipline training as biology is the largest discipline with the most data in MOPRD.”
Quote in the paperSubstantial claimMay combine two claims.
Findingnone given§ 6 ConclusionMOPRD contains paper metadata, multiple version manuscripts, review comments, meta-reviews, author's rebuttal letters, and editorial decisions of 6,578 papers.
“This dataset contains paper metadata, multiple version manuscripts, review comments, meta-reviews, author's rebuttal letters, and editorial decisions of 6,578 papers.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 4.2 DiscussionInterdisciplinary research is on the rise.
“Nowadays, interdisciplinary research is on the rise (Van Noorden, 2015), and research methods from different disciplines also continue to merge.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Van Noorden, 2015 source
Probably not a substantial claim: may not pass S2 (checkable).
Interpretationreasoning§ 4 Review comment generationMOPRD is highly suitable for the task of review comment generation.
“MOPRD as a large-scale dataset of review comment is highly suitable for the task of review comment generation.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Interpretationreasoning§ 4.1 MethodAuto-labeling can spare studies from costly handwork and is capable of multidisciplinary data.
“Auto-labeling can spare studies from costly handwork and is capable of multidisciplinary data.”
Quote in the paperNot a substantial claim- Rests on an assumption: Manual labeling is too costly for most projects in the field.
Classified as probably not a substantial claim.
Study design and conductnone given§ 4.3 ExperimentsThe records are divided randomly into training, validation, and test sets at the ratio of 8:1:1.
“These records are divided randomly into the training set, validation set, and test set at the ratio of 8:1:1.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationreasoning§ 4.2 DiscussionWith five generation models working together, the final generated review comment can be maximized to be five times the length of a traditional model.
“With five generation models working together, the final generated review comment can be maximized to be five times the length of a traditional model.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationreasoning§ 4.1 MethodThe extract-then-generate method loses an enormous amount of information from the original text, so generated review comments cannot present thorough feedback.
“This method does address the problem of length limits for text generation, but it comes with a cost because an enormous amount of information in the original text is lost in the extraction processing. This way, the generated review comments will not be able to present thorough feedback on the paper.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Premisenone given§ 4.1 MethodThere are two main problems besetting studies of review comment generation: input length and labeled data.
“There are two main problems besetting the studies of review comment generation, with input length being one and labeled data being the other.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation§ 4.1 MethodYuan et al. use an extract-then-generate method that reduces text length to within 1,024 tokens.
“Yuan et al. (2022) use an extract-then-generate method that reduces the text length to within the processing limits of 1,024 tokens.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites Yuan et al., 2022 source
Classified as probably not a substantial claim. May combine two claims.
Interpretationreasoning§ 4.2 DiscussionOur method stands out with two special designs, the modular design and the guided design.
“Our method stands out with two special designs, the modular design and the guided design.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Interpretationreasoning§ 4.2 DiscussionFor papers with a special review comment structure, the method is applicable when provided with a related labeled data source.
“For papers with a special review comment structure, our method is also applicable when provided with a related labeled data source.”
Quote in the paperNot a substantial claim- Rests on an assumption: PeerJ review comments are composed of four segments: Basic Reporting, Experimental Design, Validity of the Findings, and Additional Comments.
Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.
Interpretationreasoning§ 4.2 DiscussionCombining modular and guided design establishes a basic structure for overall generated content, enhancing logical coherence and making it more reader-friendly.
“By combining with the modular design, a basic structure is established for the overall generated content, enhancing its logical coherence and making it more reader-friendly.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationreasoning§ 4.2 DiscussionThe modular design facilitates handling long manuscripts and generating longer and more comprehensive review comments.
“the modular design facilitates the handling of long manuscripts and the generation of longer and more comprehensive review comments.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Interpretationreasoning§ 4.1.3 GenerationThe generated review comment can follow the structure of real-life review comments with higher completeness.
“The generated review comment can thus follow the structure of real-life review comments with higher completeness.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Interpretationreasoning§ 4.3.3 Comparison of generation methods over human evaluationThe high structure & logicality score occurs naturally with the special modular guided design.
“This occurs naturally with our special modular guided design.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationreasoning§ 4.2 DiscussionThrough guided design, the generated review comment of each module is appropriately guided.
“through the guided design, the generated review comment of each module is appropriately guided.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown data§ 4.3.3 Comparison of generation methods over human evaluationBoth the proposed method and the compared method are capable of generating useful and inspiring review comments.
“Both our method and the compared method are capable of generating useful and inspiring review comments, which is the core of review comment generation.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown data§ 5.1 Meta-review generationThese full peer review data enable the generation of meta-reviews in different rounds, and even using the content of manuscripts.
“These full peer review data enable the generation of meta-reviews in different rounds, and even using the content of manuscripts.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim. May combine two claims.
Study design and conductreasoning§ 4.3.4 Single-discipline training data VS multi-discipline training dataThe test data is composed of the discipline of computer science only because no computer science data are used as training data.
“To resolve overlapping between the training data and test data, the test data is composed of the discipline of computer science only as no computer science data are used as the training data.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone given§ 6 ConclusionMOPRD brings great data diversity into peer review datasets as previously they are mainly composed of data from computer science.
“It brings great data diversity into peer review datasets as previously they are mainly composed of data from computer science.”
Quote in the paperNot a substantial claim- Rests on an assumption: Most existing review comment datasets are comprised either entirely or mostly of data from the discipline of computer science only.
Classified as probably not a substantial claim. May combine two claims.
Interpretationown data§ 5.2 Editorial decision predictionComparison can be conducted to study sentimental differences across review comments from different disciplines, whereas previous analyses cannot do a multidisciplinary comparison.
“Comparison can be conducted to study sentimental differences across the review comments from different disciplines, whereas previous analyses are not able to do a multidisciplinary comparison.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable).
Interpretationreasoning§ 5.4 Scientometric analysisWith MOPRD, more profound results can be achieved on peer review scientometric analysis.
“Therefore, with MOPRD, more profound results can be achieved on peer review scientometric analysis.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Interpretationreasoning§ 5.2 Editorial decision predictionThe advantage of using MOPRD for editorial decision prediction lies in its multidisciplinary data.
“The advantage of using MOPRD for this task lies in its multidisciplinary data.”
Quote in the paperSubstantial claim
The paper relies on this without support.
PeerJ is a large general academic publisher of up to seven journals covering a wide range of research disciplines.
“PeerJ is a large general academic publisher of up to seven journals: PeerJ Analytical Chemistry , PeerJ Computer Science , PeerJ Inorganic Chemistry , PeerJ - Life and Environment , PeerJ Materials Science , PeerJ Organic Chemistry , and PeerJ Physical Chemistry . These seven journals cover a wide range of research disciplines, e.g. biology, chemistry, computer science, environment, and medicine.”
The paper relies on this without support.
PeerJ authors are given the option to present the complete peer review history along with the final published paper.
“Authors that submit papers to PeerJ are given the option to present the complete peer review history along with the final published paper to readers.”
The paper relies on this without support.
PeerJ peer review history includes initial submission, revisions, review comments, meta-reviews, author's rebuttal letters, and editorial decisions.
“For each paper, this peer review history includes the initial submission, revisions, review comments, meta-reviews, author's rebuttal letters, and editorial decisions.”
The paper relies on this without support.
Papers on PeerJ are published together with their original manuscripts available in PDF format.
“papers on PeerJ are published together with their original manuscripts available in PDF format, presenting the initial state of the papers as they are first submitted.”
The paper relies on this without support.
On PeerJ, review comments and meta-reviews are provided in plain text.
“On PeerJ, the review comments and the meta-reviews are both provided in plain text.”
The paper relies on this without support.
Review comments on PeerJ are well-structured with four segments: Basic Reporting, Experimental Design, Validity of the Findings, and Additional Comments.
“the review comments on PeerJ are well-structured with four segments: Basic Reporting, Experimental Design, Validity of the Findings, and Additional Comments.”
The paper relies on this without support.
PeerJ data are under Creative Commons Attribution License, allowing collection, remixing, adaptation, and reuse.
“what makes PeerJ stand out is the openness that comes with the Creative Commons Attribution License. With this highly accommodating license, peer review data on PeerJ are allowed to be collected, remixed, adapted, and reused as the foundation for new studies.”
The paper relies on this without support.
The input limits still exist as the maximum length of input text only reaches 16,384 tokens.
“the input limits still exist as the maximum length of the input text only reach 16,384 tokens.”
The paper relies on this without support.
The performance of attention mechanisms is compromised more and more as input length increases.
“the performance of attention mechanisms will be compromised more and more as the input length increases.”
The paper relies on this without support.
Manual labeling is too costly for most projects in the field.
“manual labeling is too costly for most projects in the field.”
The paper relies on this without support.
In peer review, it is very common for reviewers to raise questions and proposals.
“In peer review, it is very common for reviewers to raise questions and proposals in order to settle confusion in the manuscripts.”
The paper relies on this without support.
Efforts are yet to be given to questions and proposals data in review comment generation.
“However, efforts are yet to be given to these two types of data in review comment generation.”
The paper relies on this without support.
Contextual coherence and schematic structure are missing in review comments generated by pre-trained models.
“In general, contextual coherence and schematic structure are missing in the review comments generated by pre-trained models.”
The paper relies on this without support.
PeerJ review comments are composed of four segments: Basic Reporting, Experimental Design, Validity of the Findings, and Additional Comments.
“The review comments of PeerJ are composed of four segments of Basic Reporting, Experimental Design, Validity of the Findings, and Additional Comments.”
The paper relies on this without support.
Most papers have review comments in a structure similar to PeerJ's.
“Apart from PeerJ, most papers also have their review comments in a similar, if not the same, structure.”
The paper relies on this without support.
Papers in different disciplines have various writing styles and focuses.
“We notice that papers in different disciplines have various writing styles and focuses.”
The paper relies on this without support.
Beam search may lead to the generation of repetitive contents among different papers.
“The conventional beam search is not used in our method, because beam search may lead to the generation of repetitive contents among different papers.”
The paper relies on this without support.
Review comments on initial submissions are generally longer with more thorough contents.
“One reason is that review comments on the initial submissions are generally longer with more thorough contents.”
The paper relies on this without support.
Some review comments on revisions invoke the contents in the revisions and the author's rebuttal letters, adding complexity to review comment generation.
“The other reason is that some review comments on revisions invoke the contents in the revisions and the author's rebuttal letters. These data will add to the complexity of review comment generation.”
The paper relies on this without support.
Most existing review comment datasets are comprised either entirely or mostly of data from the discipline of computer science only.
“However, most existing review comment datasets are comprised either entirely or mostly of data from the discipline of computer science only.”
The paper relies on this without support.
The combination of Top-P and Top-K sampling promises more diversified and more human-like comments.
“This combination promises more diversified and more human-like comments.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
SPOT extends verification to the full complexity of frontier-level scientific research, unlike prior evaluation suites that focus on sentence-level fact checks or reproducing noisy peer-review feedback.
“It should be noted that, whereas prior evaluation suites focus on sentence-level fact checks of everyday knowledge [26, 27] or on reproducing noisy peer-review feedback [28, 29], SPOT extends verification to the full complexity of frontier-level scientific research.”
From When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research (2025) · cites this paper as [28]
Section 4.3 (p. 15) uses MOPRD manuscript–review pairs to evaluate review-comment generation, supporting the sentence’s reference to prior work on reproducing peer-review feedback. The cited paper does not characterize the peer-review feedback as noisy; its mention of grammatical errors in generated comments (Section 4.3.1, p. 16) does not establish that qualifier. The claims about sentence-level fact checks and SPOT are not attributed to this source. (LLM, conf=0.82)
Experimental — accuracy not yet measuredExisting datasets are limited to well-established computer-science conferences, with only MOPRD attempting to use multidisciplinary texts from PeerJ Publishing.
“Moreover, existing datasets are limited to wellestablished computer-science conferences, with only MOPRD attempting to use multidisciplinary texts from PeerJ Publishing.”
From The Open Review-Based (ORB) dataset: Towards Automatic Assessment of Scientific Papers and Experiment Proposals in High-Energy Physics (2023) · cites this paper as [Lin et al., 2022]
The Introduction (p. 3) says most publicly available datasets are in computer science, and Data source (p. 6) confirms that MOPRD uses PeerJ journals spanning multiple disciplines. But the claim is too absolute: the Related work table (p. 4) lists an earlier dataset covering biology, computer science, and environmental science, so the paper does not support saying existing datasets were limited to computer-science conferences or that MOPRD alone attempted multidisciplinary coverage. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThere are very few resources related to open peer-review collections.
“The authors point out a key fact that we confirmed in our own research: there are very few resources related to open peer-review collections.”
From The Open Review-Based (ORB) dataset: Towards Automatic Assessment of Scientific Papers and Experiment Proposals in High-Energy Physics (2023) · cites this paper as [Lin et al., 2022]
The Introduction (pp. 2–3) says existing open peer-review datasets often lack complete review histories and are concentrated in computer science. It does not say that there are very few open peer-review collections or resources overall; the Related work section (p. 4) lists several. The source supports a shortage of comprehensive, multidisciplinary datasets, not the broader claim. (LLM, conf=0.79)
Experimental — accuracy not yet measuredGROBID provides accurate scholarly parsing, keeps sections and citations, and is batch-ready.
“GROBID (GRO, 2008-2025) Lin et al. (2022) PDF → XML Accurate scholarly parse; keeps sections/cites; batch-ready.”
From Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future (2026) · cites this paper as Lin et al. (2022)
Section 4.1.1 (p. 10) says GROBID converts PDFs to XML while retaining the manuscripts’ original section structure; Section 3.2 (p. 7) describes its use to convert the dataset’s PDF manuscripts. The paper does not establish parsing accuracy, preservation of citations, or a batch-processing capability. The 0.986 accuracy reported in Section 4.1.1 (p. 11) is for a subsequent section-classification model, not GROBID. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Cites
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · cited 1× · see citing works
“However, the maximum input length of the previous generation of deep learning generative models, such as BART [20], is only 1,024 tokens.”
BackgroundTransformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works
Loper, E., & Bird, S. (2002) · doi:10.3115/1118108.1118117 · Cited by 3,248 · see citing works
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218 · cited 1× · see citing works
“MOPRD as a large-scale dataset of review comment is highly suitable for the task of review comment generation.”
BackgroundLongformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281 · cited 2× · see citing works
“5 MOPRD is a long-term project that will be updated and extended to cover more journals in more disciplines when adequate resources are available.”
BackgroundMethodNLTK: The Natural Language ToolkitPreprint
Loper, E., & Bird, S. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0205028 · Cited by 1,893 · see citing works
OpenNMT: Open-Source Toolkit for Neural Machine Translation
Klein, G., Kim, Y., Deng, Y., Sénellart, J., & Rush, A. M. (2017) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/p17-4012 · Cited by 1,771 · cited 1× · see citing works
“The other is to convert the output into a reader-friendly format.”
BackgroundAutomatic evaluation of summaries using N-gram co-occurrence statistics
Lin, C.-Y., & Hovy, E. H. (2003) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1073445.1073465 · Cited by 1,591 · cited 1× · see citing works
“Then, we use ROUGE [21, 22] and BARTScore [47] as the automatic metrics to evaluate the performance of different generation methods.”
MethodBig Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349 · cited 1× · see citing works
“5 MOPRD is a long-term project that will be updated and extended to cover more journals in more disciplines when adequate resources are available.”
BackgroundHierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1082 · Cited by 1,266 · cited 1× · see citing works
“The other is to convert the output into a reader-friendly format.”
BackgroundHierarchical Neural Story Generation
Fan, A., Lewis, M., & Dauphin, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.04833 · Cited by 1,242 · see citing works
A Deep Reinforced Model for Abstractive Summarization
Paulus, R., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1705.04304 · Cited by 1,182 · cited 1× · see citing works
“The other is to convert the output into a reader-friendly format.”
BackgroundInterdisciplinary research by the numbers
Van Noorden, R. (2015) · Nature · doi:10.1038/525306a · Cited by 443 · cited 1× · see citing works
“Each model can be fine-tuned within 24 hours.”
Methodvan Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999) · BMJ · doi:10.1136/bmj.318.7175.23 · Cited by 415 · see citing works
BARTScore: Evaluating Generated Text as Text Generation
Yuan, W., Neubig, G., & Liu, P. (2021) · arXiv · doi:10.48550/arxiv.2106.11520 · Cited by 353 · cited 1× · see citing works
“In this subsection, we present the technical details of our modular guided method, conduct a sanity test, compare our method with the SOTA review comment generation method of Yuan et al. [48] with MOPRD, and assess the model performance between training with single-discipline data and…”
MethodNeural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition
Soltau, H., Liao, H., & Sak, H. (2017) · Interspeech 2017 · doi:10.21437/interspeech.2017-1566 · Cited by 304 · cited 1× · see citing works
“Open-Review 2 is one example of data sources for open peer review in computer science.”
BackgroundEffect of Blinding and Unmasking on the Quality of Peer Review
van Rooyen, S., Godlee, F., Evans, S. J. W., Smith, R. W., & Black, N. A. (1998) · JAMA · doi:10.1001/jama.280.3.234 · Cited by 281 · cited 1× · see citing works
“These data, especially structured data, are of great importance to arti-ficial intelligence (AI) research [24].”
BackgroundOpen peer review: a randomised controlled trial
Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000) · British Journal of Psychiatry · doi:10.1192/bjp.176.1.47 · Cited by 261 · cited 1× · see citing works
“Open peer review is proved to attract review comments of higher quality in terms of length, helpfulness, and good manners [3, 42].”
BackgroundLopez, P. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04346-8_62 · Cited by 244 · cited 1× · see citing works
“Thus, studies based on MOPRD can be more universal and more generalized.”
BackgroundLongT5: Efficient Text-To-Text Transformer for Long Sequences
Guo, M., Ainslie, J., Uthus, D., Ontanon, S., Ni, J., Sung, Y.-H., & Yang, Y. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.55 · Cited by 230 · see citing works
A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
Kang, D., Ammar, W., Dalvi, B., Zuylen, M., Kohlmeier, S., Hovy, E., & Schwartz, R. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1149 · Cited by 171 · cited 2× · see citing works
“MOPRD is composed of paper metadata, manuscripts of the initial submission and following revisions, review comments, meta-re-views, author’s rebuttal letters, and editorial decisions of papers across various disciplines.”
BackgroundInfluentialOpen peer review: promoting transparency in open science
Wolfram, D., Wang, P., Hembree, A., & Park, H. (2020) · Scientometrics · doi:10.1007/s11192-020-03488-4 · Cited by 167 · cited 2× · see citing works
“As of December 2019, a total of 617 journals have taken up the practice of open peer review [45].”
BackgroundResultPRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization
Wen, X., Beltagy, I., Carenini, G., & Cohan, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.360 · Cited by 155 · see citing works
Defining and Characterizing Open Peer Review: A Review of the Literature
Ford, E. (2013) · Journal of Scholarly Publishing · doi:10.3138/jsp.44-4-001 · Cited by 133 · cited 1× · see citing works
“Open peer review unfreezes access to a large amount of peer review data, unlocking great opportunities for com-puter-aid peer review and further in the realization of automated scholarly paper review (ASPR) [23].”
BackgroundOpenNMT: Neural Machine Translation ToolkitPreprint
Klein, G., Kim, Y., Deng, Y., Nguyen, V., Senellart, J., & Rush, A. M. (2018) · Conference of the Association for Machine Translation in the Americas · doi:10.48550/arxiv.1805.11462 · Cited by 132 · see citing works
Can We Automate Scientific Reviewing?
Yuan, W., Liu, P., & Neubig, G. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12862 · Cited by 96 · cited 6× · see citing works
“The authors train proposition segmentation and classification models with this dataset to analyze the patterns and evaluate the utilities of argumentative propositions.”
BackgroundMethodInfluentialArgument Mining for Understanding Peer Reviews
Hua, X., Nikolov, M., Badugu, N., & Wang, L. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1219 · Cited by 86 · cited 1× · see citing works
“This dataset unlocks great opportunities for research on peer review and has already enabled several peer review studies.”
BackgroundMeasurement of Science. Study of the Development of Science as an Information Process
Nalimov, V. V., & Mulchenko, Z. M. (1971) · Defense Technical Information Center (DTIC) · Cited by 78 · see citing works
Is open peer review the fairest system? No
Khan, K. (2010) · BMJ · doi:10.1136/bmj.c6425 · Cited by 77 · cited 1× · see citing works
“However, the anonymous mechanism might in turn compromise the peer review process [17, 19, 28].”
BackgroundPeer review analyze: A novel benchmark resource for computational analysis of peer reviews
Ghosal, T., Kumar, S., Bharti, P. K., & Ekbal, A. (2022) · PLOS ONE · doi:10.1371/journal.pone.0259238 · Cited by 66 · cited 1× · see citing works
“It is constructed along with the proposal of a new task of determining whether the review comments contain requirements for the authors to act on.”
BackgroundDeepSentiPeer: Harnessing Sentiment in Review Texts to Recommend Peer Review Decisions
Ghosal, T., Verma, R., Ekbal, A., & Bhattacharyya, P. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1106 · Cited by 62 · cited 1× · see citing works
“In this section, we discuss some of the possible applications of MOPRD.”
BackgroundClosed versus open reviewing of journal manuscripts: how far do comments differ in language use?
Bornmann, L., Wolf, M., & Daniel, H.-D. (2012) · Scientometrics · doi:10.1007/s11192-011-0569-5 · Cited by 49 · cited 1× · see citing works
“Open peer review is proved to attract review comments of higher quality in terms of length, helpfulness, and good manners [3, 42].”
BackgroundDoes My Rebuttal Matter? Insights from a Major NLP Conference
Gao, Y., Eger, S., Kuznetsov, I., Gurevych, I., & Miyao, Y. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1129 · Cited by 48 · cited 1× · see citing works
“Most of the papers in PeerRead are collected together with their review comments, and some of them also have aspect specific scores such as appropriateness, clarity, and originality.”
BackgroundAutomated scholarly paper review: Concepts, technologies, and challenges
Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101830 · Cited by 37 · cited 1× · see citing works
“However, open peer review data are scattered across different platforms in various formats.”
BackgroundSelf-citationEmotionally charged text classification with deep learning and sentiment semantic
Huan, J. L., Sekh, A. A., Quek, C., & Prasad, D. K. (2021) · Neural computing & applications (Print) · doi:10.1007/s00521-021-06542-1 · Cited by 30 · cited 1× · see citing works
“The method also completely surpasses the compared method in all aspects of structure & logicality, specialty & accuracy, and usefulness & inspiration in human evaluation, with every aspect score above the mean.”
MethodResearch and Advanced Technology for Digital Libraries
Agosti, M. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-04346-8 · Cited by 29 · see citing works
MReD: A Meta-Review Dataset for Structure-Controllable Text Generation
Shen, C., Cheng, L., Zhou, R., Bing, L., You, Y., & Si, L. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.198 · Cited by 23 · cited 2× · see citing works
“Some studies [33, 35] have been conducted before on this task with other datasets.”
BackgroundThe impact of peer review on the contribution potential of scientific papers
Matsui, A., Chen, E., Wang, Y., & Ferrara, E. (2021) · PeerJ · doi:10.7717/peerj.11999 · Cited by 21 · see citing works
Augmented sentiment representation by learning context information
Hu, H., Bai, X., & Li, P. (2018) · Neural Computing and Applications · doi:10.1007/s00521-018-3698-4 · Cited by 19 · cited 1× · see citing works
“From a more detailed perspective, it also depends on the analysis of the content of multi-documents, integration of key arguments, determination of sentiment tendencies [13, 14], and inference across documents.”
BackgroundA Sentiment Augmented Deep Architecture to Predict Peer Review Outcomes
Ghosal, T., Verma, R., Ekbal, A., & Bhattacharyya, P. (2019) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl.2019.00096 · Cited by 19 · cited 1× · see citing works
“In this section, we discuss some of the possible applications of MOPRD.”
BackgroundHierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation
Deng, Z., Peng, H., Xia, C., Li, J., He, L., & Yu, P. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.555 · Cited by 19 · cited 1× · see citing works
“Editorial decision prediction is a task of predicting the decision of acceptance, revision, and rejection of a manuscript based on the review comments [5, 11].”
BackgroundProgress in Artificial Intelligence
(2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86230-5 · Cited by 17 · see citing works
Pradhan, T., Bhatia, C., Kumar, P., & Pal, S. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.11.004 · Cited by 16 · cited 1× · see citing works
“2, MOPRD mainly consists of data from five disciplines: biology, chemistry, computer science, environment, and medicine.”
BackgroundDoes My Rebuttal Matter? Insights from a Major NLP Conference
Gao, Y., Eger, S., Kuznetsov, I., Gurevych, I., & Miyao, Y. (2019) · TUbilio (Technical University of Darmstadt) · doi:10.48550/arxiv.1903.11367 · Cited by 16 · cited 1× · see citing works
“Most of the papers in PeerRead are collected together with their review comments, and some of them also have aspect specific scores such as appropriateness, clarity, and originality.”
BackgroundUncertainty-Aware Machine Support for Paper Reviewing on the Interspeech 2019 Submission Corpus
Stappen, L., Rizos, G., Hasan, M., Hain, T., & Schuller, B. W. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-2862 · Cited by 15 · cited 1× · see citing works
“What makes this dataset particularly valuable is its collection of a large number of rejected manuscripts, but unfortunately the dataset is not publicly available yet.”
BackgroundMorrison, J. (2006) · Medical Education · doi:10.1111/j.1365-2929.2006.02573.x · Cited by 14 · cited 1× · see citing works
“However, the anonymous mechanism might in turn compromise the peer review process [17, 19, 28].”
BackgroundUnderstanding and supporting anonymity policies in peer review
Nobarany, S., & Booth, K. S. (2016) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23711 · Cited by 14 · cited 1× · see citing works
“Traditional peer review [43] is based on an anonymous mechanism [30].”
BackgroundLaine, C. M. (2016) · Annals of Internal Medicine · doi:10.7326/m16-2550 · Cited by 13 · cited 1× · see citing works
“Even worse, reviewers might abuse their power to commit plagiarism (Laine, 2017).”
BackgroundAcceptance Decision Prediction in Peer-Review Through Sentiment Analysis
Ribeiro, A. C., Sizo, A., Cardoso, H. L., & Reis, L. P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86230-5_60 · Cited by 10 · cited 1× · see citing works
“In this section, we discuss some of the possible applications of MOPRD.”
BackgroundCOMPARE: A Taxonomy and Dataset of Comparison Discussions in Peer Reviews
Singh, S., Singh, M., & Goyal, P. (2021) · 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL) · doi:10.1109/jcdl52503.2021.00068 · Cited by 7 · cited 1× · see citing works
“It is composed of papers of NeurIPS, their review comments, and the number of citations.”
BackgroundAutomatic analysis of available source code of top artificial intelligence conference papers
Lin, J., Wang, Y., Yu, Y., Zhou, Y., Chen, Y., & Shi, X. (2022) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194022500358 · Cited by 7 · cited 1× · see citing works
“This is because open peer review data in computer science are most easily accessible compared to those in other disciplines.”
BackgroundSelf-citationReAct: A Review Comment Dataset for Actionability (and more)
Choudhary, G., Modani, N., & Maurya, N. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-91560-5_24 · Cited by 6 · cited 1× · see citing works
“Papers of biology, computer science, and environment are studied with traditional machine learning technologies to understand the impact of peer review process on papers, including the acceptance timeline, potential contributions, manuscript modifications.”
Background
Flagged first, then by citations
Research works printed but not linked: 1
- Ware M, Mabe M (2015) The STM Report: An overview of scientific and scholarly journal publishing, 4th edn. International Association of Scientific, Technical and Medical Publishers
Cited by
Sorted by citations
Large language models for automated scholarly paper review: A survey
Zhuang, Z., Chen, J., Xu, H., Jiang, Y., & Lin, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103332 · Cited by 34 · see citing works
“To address this limitation, methods such as the extract-then-generate method (Yuan et al., 2022) and modular guided method (Lin et al., 2023b) have been proposed.”
Self-citationDefining quality in peer review reports: a scoping review
Sizo, A., Lino, A. D. P., Rocha, Á., & Reis, L. P. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02435-0 · Cited by 26 · see citing works
Wu, W., Zhang, C., & Zhao, Y. (2025) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70005 · Cited by 26 · see citing works
“Lin et al. (2023b) proposed MOPRD, a multidisciplinary open peer-review dataset for automatic generation of structured peer reviews, providing interdisciplinary research for peer review.”
LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
Du, J., Wang, Y., Zhao, W., Deng, Z., Liu, S., Lou, R., Zou, H. P., Narayanan Venkit, P., Zhang, N., Srinath, M., et al. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.292 · Cited by 23 · see citing works
Large language models in peer review: challenges and opportunities
Sun, Z. (2025) · Scientometrics · doi:10.1007/s11192-025-05440-w · Cited by 15 · see citing works
“…datasets, such as MOPRD, Open Review-Based datasets, and withdrawn paper datasets from arXiv, which provide a broader scope of peer review data across various fields, also exist (Liang et al., 2024b; Lin et al., 2023; Szumega et al., 2023; Tan et al., 2024; Zhang & Abernethy, 2025).”
PolitePEER: does peer review hurt? A dataset to gauge politeness intensity in the peer reviews
Bharti, P. K., Navlakha, M., Agarwal, M., & Ekbal, A. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09662-3 · Cited by 12 · see citing works
When AI Co-Scientists Fail: SPOT-a Benchmark for Automated Verification of Scientific Research
Son, G., Hong, J., Fan, H., Nam, H., Ko, H., Lim, S., Song, J., Choi, J., Paulo, G., Yu, Y., et al. (2025) · arXiv · doi:10.48550/arxiv.2505.11855 · Cited by 5 · see citing works
“It should be noted that, whereas prior evaluation suites focus on sentence-level fact checks of everyday knowledge [26, 27] or on reproducing noisy peer-review feedback [28, 29], S POT extends verification to the full complexity of frontier-level scientific research.”
Self-citationClaimBrush: A Novel Framework for Automated Patent Claim Refinement Based on Large Language Models
Kawano, S., Nonaka, H., & Yoshino, K. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825172 · Cited by 4 · see citing works
“As research similar to our study, there is research on the proofreading task of scientific papers, which is approached from various aspects such as grammatical error correction and rewriting [15], [16], and review comment generation [17]– [20].”
Wei, S., Lai, Z., & Shi, X. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126987 · Cited by 2 · see citing works
Self-citationCo-Reviewer: can AI review like a human? An agentic framework for LLM-human alignment in peer review
Bharti, P. K., Dalal, V., & Panchal, M. (2026) · Scientometrics · doi:10.1007/s11192-026-05557-6 · Cited by 2 · see citing works
LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing
Du, J., Wang, Y., Zhao, W., Deng, Z., Liu, S., Lou, R., Zou, H. P., Venkit, P. N., Zhang, N., Srinath, M., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.16253 · Cited by 2 · see citing works
“…in automatically generating peer reviews, such as predicting scores (Li et al., 2020; Zhou et al., 2024; Wang et al., 2020; Deng et al., 2020) and writing reviews (Gao et al., 2024; Wang et al., 2020; Yuan et al., 2022; Liu and Shah, 2023) and meta-reviews (Li et al., 2023; Lin et al., 2023b).”
BackgroundChang, Y., Li, Z., Zhang, H., Kong, Y., Wu, Y., So, H. K.-H., Guo, Z., Zhu, L., & Wong, N. (2025) · Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2025.emnlp-main.790 · Cited by 2 · see citing works
Can AI Be a Good Peer Reviewer? A Survey of Peer Review Process, Evaluation, and the Future
Wu, S., Jiang, O., Zhao, Y., Hu, T., Ma, Y., Zhang, K., Patwardhan, M., & Cohan, A. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1504 · Cited by 1 · see citing works
Szumega, J., Bougueroua, L., Gkotse, B., Jouvelot, P., & Ravotti, F. (2023) · arXiv · doi:10.48550/arxiv.2312.04576 · Cited by 1 · see citing works
“A second important reference is the Multidisciplinary Open Peer Review Dataset (MOPRD) by [Lin et al., 2022], available online 8 .”
BackgroundUnpacking the predictive power of peer review under scientific disruption: Evidence from AI research
Zhang, Y.-J., Jiang, Y.-F., Yan, P.-W., Jiang, Z.-R., Lin, C.-X., He, G., & Liu, X.-Z. (2026) · J. Informetrics · doi:10.1016/j.joi.2026.101811
SurveyReview: A Reviewer-Aligned Benchmark for Survey Evaluators
Zhang, Y., Wang, Y., Zhang, F., Zhao, R., Li, J., Tang, J., & Zhang, J. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817505
RECO-Rank: A Compensatory Multi-Evidence Ranking Model for Cross-Domain Reviewer RecommendationPreprint
Chen, J., Xu, J., & Fang, Z. (2026) · Research Square · doi:10.21203/rs.3.rs-9558981/v1
Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review
Yuan, S., Zhang, X., Ni, W., Xiao, T., Di, S., Zhu, J., Jiang, Z., Tan, R., Chen, L., & Ren, K. (2026) · arXiv · doi:10.48550/arxiv.2609.05947
“PeerSum (Li, Qi, and Lau 2022), MOPRD (Lin et al. 2023), and resources for peer-review argumentation (Fromm et al. 2021) support review summarization, multi-stage review records, and discourse analysis, respectively.”
- Authors
- Lin, J., Song, J., Zhou, Z., Chen, Y., & Shi, X.
- Affiliations
- Xiamen University (CN)
- Corresponding author
- Shi, Xiaodong
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Volume
- 35
- Issue
- 34
- Pages
- 24191-24206
- ISSN
- 0941-0643 (ISSN-L)
- arXiv
- 2212.04972
- OpenAlex
- W4386986268
- Cited by
- 11
- References
- 42
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 1 · 2023: 1
- FWCI
- 9.80
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Neural Computing and Applications (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- open peer review, scientometrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
