CoChat
Sign Up
PreprintCited by 3scientometrics and bibliometrics researchAcademic Publishing and Open AccessPublishing and Scholarly Communication

The Open Review-Based (ORB) dataset: Towards Automatic Assessment of Scientific Papers and Experiment Proposals in High-Energy Physics

Open access
Szumega, Jaroslaw, Bougueroua, Lamine, Gkotse, Blerina, Jouvelot, Pierre, Ravotti, Federico·2023·arXiv
DOI 10.48550/arxiv.2312.04576arXiv 2312.04576
Open this paper in CoChat — free accountOpen PDF

Abstract

With the Open Science approach becoming important for research, the evolution towards open scientific-paper reviews is making an impact on the scientific community. However, there is a lack of publicly available resources for conducting research activities related to this subject, as only a limited number of journals and conferences currently allow access to their review process for interested parties. In this paper, we introduce the new comprehensive Open Review-Based dataset (ORB); it includes a curated list of more than 36,000 scientific papers with their more than 89,000 reviews and final decisions. We gather this information from two sources: the OpenReview.net and SciPost.org websites. However, given the volatile nature of this domain, the software infrastructure that we introduce to supplement the ORB dataset is designed to accommodate additional resources in the future. The ORB deliverables include (1) Python code (interfaces and implementations) to translate document data and metadata into a structured and high-level representation, (2) an ETL process (Extract, Transform, Load) to facilitate the automatic updates from defined sources and (3) data files representing the structured data. The paper presents our data architecture and an overview of the collected data along with relevant statistics. For illustration purposes, we also discuss preliminary Natural-Language-Processing-based experiments that aim to predict (1) papers' acceptance based on their textual embeddings, and (2) grading statistics inferred from embeddings as well. We believe ORB provides a valuable resource for researchers interested in open science and review, with our implementation easing the use of this data for further analysis and experimentation. We plan to update ORB as the field matures as well as introduce new resources even more fitted to dedicated scientific domains such as High-Energy Physics.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces ORB, an Open Review-Based dataset of scientific submissions and peer reviews collected from OpenReview.net and SciPost.org, with supporting Python interfaces, ETL processes, and an ontology. It also reports preliminary NLP experiments predicting paper acceptance and grading statistics from textual embeddings.

Key findings

  • ORB contains 36,949 unique submissions, 2,081 submissions with multiple revisions, 27,510 submissions with reviews, and 92,879 reviews in total (Table 1); the abstract describes this as more than 36,000 papers and more than 89,000 reviews.
  • The dataset is built from OpenReview.net (34,030 unique submissions, 85,470 reviews) and SciPost.org (2,919 unique submissions, 7,409 reviews), with SciPost providing structured reports and six separate grades for Validity, Significance, Originality, Clarity, Formatting, and Grammar.
  • Preliminary acceptance prediction on a SciPost subset of 4,917 elements achieved accuracy 0.59 with a Support Vector Classifier and 0.55 with k-Nearest Neighbors.
  • Preliminary grading-statistics prediction on over 16,000 submissions from NeurIPS, ICLR, and UAI yielded MAE values of 0.87 for final score, 0.78 for score variance, 0.40 for confidence, and 0.30 for confidence variance.

Methods

  • Dataset construction: data were collected from OpenReview.net via its API and from SciPost.org via a custom HTML scraper, then organized through an ETL process into OrbRaw* interfaces and Orb* dataclasses; the paper also introduces the CEDIgaR methodology for continuous dataset extension.
  • Data: the current ORB version includes submissions, reviews, decisions, ratings, and metadata from OpenReview.net and SciPost.org, with full-text papers referenced by PDF URLs rather than stored directly.
  • Preliminary experiments: a Doc2Vec model was trained on the scientific_papers dataset of 200,000 ArXiv and PubMed documents with 512-dimensional vectors; acceptance was predicted using SVC with RBF kernel and kNN (n_neighbors = 10), and grading statistics were predicted using a keras_tuner-tuned neural network evaluated by MAE.

Limitations

  • Stated: full texts are delivered as PDF URLs only, not stored in ORB, and PDF extraction is not addressed because of segmentation and multimodal concerns.
  • Stated: although no personal data were stored in the original sources, ORB contains paper titles and PDF URLs, so author identity could be indirectly inferred, raising ethical privacy concerns; the paper also notes that large-scale NLP analysis may require more informed consent.
  • Stated: the approximately 89,000 reviews were copied unchanged and were not semantically examined for offensive content; the preliminary experiments are also limited, with acceptance prediction affected by SciPost versioning and grading prediction not accounting for author-reviewer discussions.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2312.04576

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

6 of 23 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    There are very few resources related to open peer-review collections.

    “The authors point out a key fact that we confirmed in our own research: there are very few resources related to open peer-review collections.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [Lin et al., 2022] source

      The Introduction (pp. 2–3) says existing open peer-review datasets often lack complete review histories and are concentrated in computer science. It does not say that there are very few open peer-review collections or resources overall; the Related work section (p. 4) lists several. The source supports a shortage of comprehensive, multidisciplinary datasets, not the broader claim. (LLM, conf=0.79)

      Experimental — accuracy not yet measured
  • Premisecitation

    Existing peer-review datasets are used in the vast majority to predict acceptance and scores or citations.

    “They compare existing peer-review datasets and their applications; in the vast majority, the existing items are used to predict acceptance and scores ( [Gao et al., 2019, Kang et al., 2018]) or citations ( [Plank and van Dalen, 2019]).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [Gao et al., 2019, Kang et al., 2018] source

      The available passages from our copy are bibliographic records, not the paper’s discussion of peer-review datasets (no section or page is available). Searches for dataset applications and prediction of acceptance or scores returned no relevant passage. The citation to Plank and van Dalen, rather than Gao et al., is attached to the separate citations claim. (LLM, conf=0.70)

      Experimental — accuracy not yet measured
    • cites [Plank and van Dalen, 2019]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Findingown data

    ORB data is gathered from OpenReview.net and SciPost.org.

    “We gather this information from two sources: the OpenReview.net and SciPost.orgwebsites.”
    • Also stated in: § 1 Introduction · plain
      “In this first version, ORB is populated with the publicly available data provided by the OpenReview.net and SciPost.orgplatforms.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    As part of the ORB infrastructure, implementations of OrbRaw* interfaces are provided for the OpenReview.net and SciPost.org data collections.

    “As the part of the ORB infrastructure, we provide implementations of OrbRaw* interfaces for the OpenReview.net and SciPost.orgdata collections.”
    Quote in the paperSubstantial claim
  • Findingnone given

    The ORB project produced data files containing instances of all current collected submissions and reviews, structured according to ORB principles.

    “The data files containing the instances of all the current collected submissions and reviews, structured according to the ORB principles.”
    Quote in the paperSubstantial claim
  • May combine two claims. 26
  • Classified as probably not a substantial claim. 45
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation

    Existing datasets are limited to well-established computer-science conferences, with only MOPRD attempting to use multidisciplinary texts from PeerJ Publishing.

    “Moreover, existing datasets are limited to wellestablished computer-science conferences, with only MOPRD attempting to use multidisciplinary texts from PeerJ Publishing.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [Lin et al., 2022] source

      The Introduction (p. 3) says most publicly available datasets are in computer science, and Data source (p. 6) confirms that MOPRD uses PeerJ journals spanning multiple disciplines. But the claim is too absolute: the Related work table (p. 4) lists an earlier dataset covering biology, computer science, and environmental science, so the paper does not support saying existing datasets were limited to computer-science conferences or that MOPRD alone attempted multidisciplinary coverage. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Premiseown data

    PeerRead's last update in its GitHub repository dates back to May 2018, and its content is limited to a few conferences before 2018.

    “Despite the fact that this dataset was supposed to be released periodically, the last update available in its GitHub repositoryfoot_4 dates back to May 2018.The content is currently limited, based on a few conferences before 2018.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [Kang et al., 2018] source

      The Introduction says the authors planned periodic releases, and §2.3 lists five sections in version 1.0, covering conferences through 2017 and arXiv submissions through 2017. The paper does not establish that May 2018 was the last GitHub update or that the dataset remained limited to those sections at the time of the citing sentence. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The number of publishers and journals applying Open Peer Review increased from 20 publishers and 174 journals in 2018 to 38 publishers and 617 journals in 2020.

    “20 publishers and 174 journals in 2018 compared to 38 publishers and 617 journals in 2020, according to [Wolfram et al., 2019] and [Wolfram et al., 2020]”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [Wolfram et al., 2019]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [Wolfram et al., 2020] source

      The Data verification and cleaning section reports 20 publishers and 174 OPR journals in a dataset collected in 2018, and 38 publishers and 617 OPR journals in the expanded dataset. It dates the latter count to December 2019, not 2020. (LLM, conf=0.90)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The scientific_papers dataset contains abstracts and texts of 200,000 instances of ArXiv and PubMed open-access documents.

    “which contains abstracts and texts of 200,000 instances of ArXiv and PubMed open-access documents”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [Cohan et al., 2018] source

      Section 5 (Data) says the paper introduces arXiv and PubMed datasets of scientific papers, using their abstracts as summaries. But Table 1 reports 215,000 arXiv documents and 133,000 PubMed documents—348,000 combined—not 200,000 instances. The cited passages do not establish that all documents are open-access. (LLM, conf=0.90)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The fields of the OrbRaw* classes reflect the original information stored on the data portals.

    “The fields of these classes reflect the original information stored on the data portals.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    All SciPost data is available only as a website; obtaining relevant information requires extracting data from every single HTML page and representing it as a data object.

    “All the data is available only as a website; in order to obtain relevant information, the data needs to be extracted from every single HTML page and represented as a data object.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingcitation

    The ORB data architecture is implemented in the framework of ETL processes.

    “To maintain clarity and extensibility of our solution, the data architecture is implemented in the framework of ETL (Extract, Transform, Load) processes, a concept designed for data extraction, processing and exchange [Vassiliadis, 2009] and widely used in data modeling and warehousing [Trujillo and Luján-Mora, 2003].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [Vassiliadis, 2009] source
    • cites [Trujillo and Luján-Mora, 2003] source

  • Classified as probably not a substantial claim. May combine two claims.

    Findingnone given

    Submissions and reviews from different online libraries, OPR journals, and submission portals host various data and metadata; their designs usually follow similar ideas, but details often do not correspond when comparing two sources.

    “We observed that the submissions and reviews from different online libraries, OPR journals and submission portals host various data and metadata.Their designs usually follow similar ideas, but details often do not correspond to one another when comparing two sources.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingreasoning

    The ORB project introduced the CEDIgaR procedure for continuous data integration in extensible datasets.

    “To achieve such a goal, we introduced the new CEDIgaR procedure that can be followed in order to perform continuous data integration in any extensible dataset.”
    • Also stated in: § 3.4 Delivered resources · plain
      “In addition to these concrete artefacts, we introduced the concept of CEDIgaR, a methodology specifying the key steps required for designing a dataset subject to continuous development.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    SciPost lacks any API allowing data access, unlike OpenReview.net.

    “However, the major disadvantage, in comparison to OpenReview.net, is the lack of any API allowing data access.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Obtaining data from OpenReview requires diligent configuration of the client and its search parameters.

    “The process of obtaining data requires a diligent configuration of the client and of its search parameters.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The ORB dataset includes more than 36,000 scientific papers/submissions and more than 89,000 reviews and final decisions.

    “it includes a curated list of more than 36,000 scientific papers with their more than 89,000 reviews and final decisions.”
    • Also stated in: § 3 ORB dataset construction · plain
      “ORB stores currently over 36,000 submissions (including corrected re-submissions) and 89,000 reviews.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    SciPost's total scoring consists of six separate grades describing each submission's Validity, Significance, Originality, Clarity, Formatting, and Grammar.

    “The total scoring consists of six separate grades describing each submission's Validity, Significance, Originality, Clarity, Formatting and Grammar.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    SciPost's reports are separated into four sections: Strengths, Weaknesses, Report comments, and Requested changes.

    “They are separated into four sections: (1) Strengths, (2) Weaknesses, (3) Report comments and (4) Requested changes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Both MDPI and PeerJ support Open Review Policy principles, and PeerJ publishes a peer-review history since February 2023.

    “Having reviewed the Open Review Policy (ORP) notices from MDPI and PeerJfoot_2 , we found that both publishers support the ORP principles while PeerJ also publishes a peer-review history since February 2023.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Every OpenReview Note is represented as a Python dictionary whose nested keys are set only if values exist, resulting in missing keys when parsing data.

    “Every”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    ORB deliverables include Python code (interfaces and implementations), an ETL process, and data files.

    “The ORB deliverables include (1) Python code (interfaces and implementations) to translate document data and metadata into a structured and high-level representation, (2) an ETL process (Extract, Transform, Load) to facilitate the automatic updates from defined sources and (3) data files representing the structured data.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The ORB dataset contains scientific submissions and their reviews from established Computer Science conferences (OpenReview.net) and from open journals in natural sciences, especially High-Energy Physics (SciPost.org).

    “It contains scientific submissions and their reviews from established conferences in the field of Computer Science (OpenReview.net)and from open journals in the field of natural sciences, especially High-Energy Physics (SciPost.org).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    The authors chose licensing under the Creative Commons Attribution 4.0 International (CC BY 4.0) License for ORB code and other deliverables.

    “To facilitate usage of ORB, its relevant code and other deliverables, we chose licensing under the Creative Commons Attribution 4.0 International (CC BY 4.0) License.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The ORB dataset and the used data sources contain the papers' titles and URLs to PDF.

    “However, the ORB dataset, as also the used data sources, contain the papers' titles and URLs to PDF.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The papers' full-texts in the data sources are delivered as PDF files.

    “Firstly, the papers' full-texts are delivered as PDF files.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The neural network prediction for grading statistics may be off by around 0.8 per final score and 0.4 per confidence.

    “These values show that the network prediction may be off by around 0.8 per final score and 0.4 per confidence.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The code is publicly available on CERN Gitlab and licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) License.

    “Additionally (as specified in the previous section), the code is publically available on CERN Gitlab and licensed under Creative Commons Attribution 4.0 International (CC BY 4.0) License.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The ORB dataset is built upon OpenReview (open access, MIT License specified for API client) and SciPost (open access, Creative Commons Attribution 4.0 International License).

    “We are building the ORB dataset upon available services and data: OpenReview (open access, MIT License 16 specified for API client) and SciPost (open access, Creative Commons Attribution 4.0 International (CC BY 4.0) License 17 ).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The concrete results of the work (datafiles, dataset design, interfaces and dataclasses implementation incorporated into ETL processes, ontology) are shared publicly on the CERN GitLab server.

    “The concrete results of our work (datafiles, dataset design, interfaces and dataclasses implementation incorporated into ETL processes, ontology) are shared publicly for inspection, usage and discussions regarding further development on the CERN GitLab server at https://gitlab.cern.ch/irrad/ orb-dataset.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The provided data, especially reviews, are in an unchanged form, being straight copies from the original provider sources.

    “Finally, and still in the realm of ethics, the provided data (especially reviews) are in an unchanged form, being straight copies from those present in the original provider sources.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    The collection process aims to unify data, and ORB* classes reorganize the data format to fit the schema described in Section A.2.

    “However, the collection process is aiming to unify data, therefore ORB* classes are reorganizing the data format in order to fit the schema described in the Section A.2.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The collection process is implemented as the ETL process, allowing potential users to re-run it and update the dataset with a Python script.

    “The collection process is implemented as the ETL process (Extract -Transform -Load), allowing for potential users to re-run it and update the dataset with Python script.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    All stages of data collection (collected raw information, OrbRaw* instances and target Orb* instances) can be stored for further use.

    “All stages of data collection (collected raw information, OrbRaw* instances and target Orb* instances) can be stored for further use.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    The dataset instances are collected from two sources: OpenReview.net and SciPost.org.

    “The dataset instances are collected from two sources, however, the methods for data collection differs in these cases:”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Findingnone given

    The dataset delivers the Python implementation of OrbRaw* interfaces and Orb* dataclasses that represent the Open Peer Review process, with history and changes when available.

    “Besides included datafiles, the dataset delivers the Python implementation of OrbRaw* interfaces and Orb* dataclasses that represent the Open Peer Review process, with history and changes when available.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The ORB dataset uses OpenReview.Net and SciPost.org as sources of Open Peer Review data.

    “Currently it uses the OpenReview.Net and SciPost.org as sources of Open Peer Review data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    No semantic examination of the about 89,000 reviews for the presence of offensive content was performed.

    “Therefore, no semantic examination of the about 89,000 reviews for the presence of offensive content was performed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Orb* dataclasses are a highly structured generalization supposed to unify different data sources.

    “Orb* dataclasses are highly structured generalization that is supposed to unify different data sources (see Figure A.1).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing). May combine two claims.

    Premisecitation

    All online content published on SciPost.org is licensed under Creative Commons Attribution 4.0 International (CC BY 4.0), allowing redistribution, transformation, and building upon existing material.

    “As the”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites SciPost Terms and Conditions

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    About 90% of the world's data is particularly challenging to integrate.

    “Different sources of data will differ not only in content but also in their structure or have no defined structure at all -similarly to about 90% of the world's data that is particularly challenging to integrate ( [Mehmood et al., 2019]).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [Mehmood et al., 2019] source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    MReD is annotated by hired professional annotators, so the annotation process seems to be done manually.

    “The proposed dataset is also annotated by hired professional annotators; as such, the annotation process seems to be done manually.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [Shen et al., 2022] source

  • May combine two claims.

    Premisecitation

    Numerous samples of peer reviews and representative cross-sections of scores are critical to minimize bias and avoid high-frequency pattern generation in machine-learning-based review generation.

    “numerous samples of peer reviews and representative cross-sections of scores are critical to minimize any bias possibly existing in the documents and avoid high-frequency pattern generation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [Yuan et al., 2022] source

  • May combine two claims.

    Interpretationnone given

    The ORB dataset represents a first step into providing an initial set of publicly available scientific reviews and a computing infrastructure able to accommodate future related sources.

    “The new Open Review-Based (ORB) dataset introduced herefoot_0 represents a first step into providing both an initial set of publicly available scientific reviews and a computing infrastructure able to accommodate the future introduction of new related sources.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The Open Peer Review movement will produce large and publicly available resources whose content is related to the experiment proposals the RADNEXT project wants to assess.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Data integration provides better accessibility, especially for data from non-cooperating sources.

    “Such data integration provides, in return, better accessibility, especially in case of data from non-cooperating sources [Stupnikov, 2021].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [Stupnikov, 2021] source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    MReD data is retrieved from the ICLR 2018-2021 conferences.

    “The data is retrieved from the ICLR 2018-2021 conferences.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [Shen et al., 2022] source

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    The Open Peer Review process is at least partially transparent and subjected to broader community assessment, therefore improving the quality and objectivity of reviews.

    “The Open Peer Review process is at least partially transparent and subjected to a broader community's assessment -therefore improving the quality of the reviews and their objectivity.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Access to Elsevier's Peer Review Workbench dataset is granted only when a proper research proposal is submitted beforehand.

    “However, the official documentation of this datasetfoot_3 states that the access is granted only when a proper research proposal is submitted beforehand.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [Williams, 2021]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Analysis of the conceptual schema of each source results in better data integration and enhanced reusability.

    “However, the analysis of the conceptual schema of each source results in better data integration and enhanced reusability [Castano and De Antonellis, 2001].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [Castano and De Antonellis, 2001] source

  • Classified as probably not a substantial claim.

    Premisenone given

    There is a lack of publicly available resources for research on open scientific-paper reviews, as only a limited number of journals and conferences allow access to their review process.

    “However, there is a lack of publicly available resources for conducting research activities related to this subject, as only a limited number of journals and conferences currently allow access to their review process for interested parties.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    With a large enough OPR-centered dataset, it will be possible to conduct related research and build NLP models for a more objective document-evaluation process or for checking bias in non-OPR comments.

    “With a large enough OPR-centered dataset, it will be possible to conduct related research and build NLP models for a possibly more objective document-evaluation process or even checking the presence of possible bias in non-OPR comments.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The Open Peer Review movement will produce large and publicly available resources whose content is related to the experiment proposals the RADNEXT project wants to assess.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Authors still have the possibility to opt out of making reviews public.

    “However, authors have still the possibility to opt-out of making reviews public.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisenone given

    All three datasets are strongly pre-processed by text segmentation, exclusions of parts of submissions, and data annotations; annotation is done with external tools or by human experts and is not an inherent feature derived from source data.

    “All three datasets are strongly pre-processed by text segmentation, exclusions of parts of the submissions and data annotations.The annotation is done with external tools or by human experts; it is not an inherent feature derived from the source data.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Most available datasets for summarization were either focused on the news category or contained only a few hundred input papers.

    “The authors point out that most of the available datasets for summarization were either focused on the news category or contained only a few hundred input papers.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [Shen et al., 2022] source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The Meta-Review Dataset (MReD) consists of 7,089 meta-reviews and was used for structure-controllable text generation.

    “The authors created a new dataset consisting of 7,089 meta-reviews that was used for structure-controllable text generation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [Shen et al., 2022] source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The ORB software infrastructure is designed to accommodate additional resources in the future.

    “the software infrastructure that we introduce to supplement the ORB dataset is designed to accommodate additional resources in the future.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    An important factor influencing OpenReview grades is the impact of possible discussions between authors and reviewers.

    “For instance, an important factor is the impact of the possible discussions between authors and reviewers.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    In SciPost data, the corrected text between successive versions may not change too much.

    “There, the corrected text between successive versions may not change too much”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The authors avoided unnecessary preprocessing that strips information in order to deliver all relevant data properties.

    “We avoided the unnecessary preprocessing that strips information in order to deliver all relevant data properties.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The naming information in ORB could be indirectly inferred or obtained via PDF extraction and analysis, possibly posing ethical privacy concerns.

    “This naming information could thus be indirectly inferred or via PDF extraction and analysis, possibly posing ethical privacy concerns.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    To the best of the authors' knowledge, ORB is the first dataset containing experimental HEP-related peer review data.

    “To the best of our knowledge, ORB is the first dataset containing experimental HEP-related peer review data.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Existing datasets are mostly limited to the discipline of Computer Science.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The poor acceptance prediction result may be due to versioning in SciPost data.

    “This somewhat poor result may be due to one inherent feature of SciPost data: versioning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    One possible reason for the wide error ranges in grading statistics prediction is that the experiment does not take into account that OpenReview grades are often influenced by elements other than the submitted text.

    “One possible reason for these somewhat wide error ranges is that this experiment does not take into account that the OpenReview grades are often influenced by elements other than the mere submitted text.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The Open Peer Review process provides the possibility to link submissions and reviews with indirect metadata, such as information associated with authors or with a preprint.

    “The Open Peer Review process provides interesting possibility to link submissions and reviews with indirect metadata [Wang et al., 2023], e.g., the information that can be associated with authors, or with the paper itself that was published as a preprint even before submission to journal/conference supporting Open Peer Review.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [Wang et al., 2023]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

59 of 59

  • The paper relies on this without support.

    The number of already evaluated RADNEXT submissions and corresponding reviews is too limited to train a machine-learning model.

    “Unfortunately, the number of the already evaluated RADNEXT submissions and corresponding reviews is limited for now, and as such, this data wouldn't be sufficient to ensure the proper training of any machine-learning-based model.”

  • The paper relies on this without support.

    The Open Peer Review movement will produce large and publicly available resources whose content is related to the experiment proposals the RADNEXT project wants to assess.

    “This radical transformation of the scientific assessment of research papers by conference and journal editors, moving from the still prevalent use of blind and secretive reviews to an open approach, is bound to produce large and publicly available resources (papers, reviews, scores, etc.) whose content is clearly related to the experiment proposals we want to help assess within the RADNEXT project”

  • The paper relies on this without support.

    Unification of heterogeneous data is a complicated task.

    “Unification of heterogeneous data is a complicated task.”

  • The paper relies on this without support.

    The Open Peer Review movement is still in its infancy.

    “since the OPR movement is still in its infancy, it is currently difficult to identify sources that can provide generally accessible reviewing information.”

  • The paper relies on this without support.

    Existing datasets are mostly limited to the discipline of Computer Science.

    “Since (to our best knowledge) the existing dataset are mostly limited to the discipline of Computer Science, we decided to build our own resource that contains data of Open Review process in HEP (see SciPost datasource).”

  • The paper relies on this without support.

    No personal data were stored in the original data sources during the peer-review process, neither for the authors nor for the reviewers.

    “Secondly, no personal data were stored in the original data sources during the peer-review process (neither for the authors nor for the reviewers).”

  • The paper relies on this without support.

    The OpenReview and SciPost websites' guidelines make it clear to contributors that submissions and reviews are envisioned to be public.

    “The OpenReview and Scipost websites already deal with some of the these concerns: the submissions and their reviews are envisioned to be public, a fact made quite clear to the contributors by the sites' guidelines.”

  • The paper relies on this without support.

    In the case of a rejected paper, agreeing to still take part in the open-access process is up to the authors.

    “For instance, already now, in the case of a rejected paper, agreeing to still take part in the open-access process is up to the authors (an additional opt-in decision).”
8 of 8

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Barbara Plank and Reinard van Dalen. Citetracked: a longitudinal dataset of peer reviews and citations. In 4th Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2019), pages 116-122. CEUR Workshop Proceedings, 2019.
  2. Linda Williams. New dataset offers unique insights into peer review, Sep 2021. URL https://www.elsevier. com/connect/new-dataset-offers-unique-insights-into-peer-review.
  3. Dietmar Wolfram, Peiling Wang, and Hyoungjoo Park. Open peer review: The current landscape and emerging models. 2019.

Cited by

Sorted by citations

Related

Authors
Szumega, J., Bougueroua, L., Gkotse, B., Jouvelot, P., & Ravotti, F.
Affiliations
  • Université Paris-Panthéon-Assas (FR)
  • Centre de Recherche en Informatique (FR)
  • European Organization for Nuclear Research (CH)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2312.04576
arXiv
2312.04576
OpenAlex
W4389598890
Cited by
1
References
17
Citations by year
2026: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • HAL (Le Centre pour la Communication Scientifique Directe) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • CERN (101008126)
  • European Commission (HORIZON2020, 101008126)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
Keywords
open science, peer review, particle physics, structured data extraction, ETL pipeline, text embeddings

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review