CoChat
Sign Up
Cited by 167Academic Publishing and Open AccessPublishing and Scholarly Communicationscientometrics and bibliometrics research

Open peer review: promoting transparency in open science

Open access
Wolfram, Dietmar, Wang, Peiling, Hembree, Adam, Park, Hyoungjoo·2020·Scientometrics
DOI 10.1007/s11192-020-03488-4
Open this paper in CoChat — free accountOpen PDF

Abstract

Open peer review (OPR), where review reports and reviewers’ identities are published alongside the articles, represents one of the last aspects of the open science movement to be widely embraced, although its adoption has been growing since the turn of the century. This study provides the first comprehensive investigation of OPR adoption, its early adopters and the implementation approaches used. Current bibliographic databases do not systematically index OPR journals, nor do the OPR journals clearly state their policies on open identities and open reports. Using various methods, we identified 617 OPR journals that published at least one article with open identities or open reports as of 2019 and analyzed their wide-ranging implementations to derive emerging OPR practices. The findings suggest that: (1) there has been a steady growth in OPR adoption since 2001, when 38 journals initially adopted OPR, with more rapid growth since 2017; (2) OPR adoption is most prevalent in medical and scientific disciplines (79.9%); (3) five publishers are responsible for 81% of the identified OPR journals; (4) early adopter publishers have implemented OPR in different ways, resulting in different levels of transparency. Across the variations in OPR implementations, two important factors define the degree of transparency: open identities and open reports. Open identities may include reviewer names and affiliation as well as credentials; open reports may include timestamped review histories consisting of referee reports and author rebuttals or a letter from the editor integrating reviewers’ comments. When and where open reports can be accessed are also important factors indicating the OPR transparency level. Publishers of optional OPR journals should add metric data in their annual status reports.

AI snapshot

Written from the paper’s full text.

TL;DR

This study provides the first comprehensive investigation of open peer review (OPR) adoption, identifying 617 OPR journals as of 2019 and analyzing how publishers implement open identities and open reports. It finds steady growth since 2001, rapid growth since 2017, and wide variation in transparency, with only a minority requiring both open identities and open reports.

Key findings

  • OPR adoption grew from 38 journals in 2001 to 617 journals by December 2019, with the total more than doubling since 2017; adoption is concentrated in Medical and Health Sciences (258 journals, 41.8%) and Natural Sciences (235, 38.1%), together 79.9% of identified OPR journals.
  • Five publishers account for 500 of 617 OPR journals (81.0%): MDPI (204, 33.0%), SDI (111, 18.0%), BioMed Central (70, 11.3%), Frontiers Media S.A. (64, 10.4%), and Kowsar (51, 8.3%).
  • Most OPR journals allow or require open identities: 608 (98.5%) either require reviewers to identify themselves (268, 43.4%) or allow reviewers to choose (340, 55.1%); 536 (86.9%) either require open reports (274, 44.4%) or allow authors/editors to choose (259, 42.3%).
  • Only 189 journals (30.6%) require both open identities and open reports; 24 of 38 publishers (63.2%) are based in Europe and account for 445 of 617 titles (72.1%).

Methods

  • The authors identified OPR journals using multiple sources, including DOAJ, Transpose, publisher websites, newsletters, and peer-review-related sites, then verified each journal’s website and early issues to confirm whether it published open identities or open reports.
  • Inclusion required at least one of two core OPR elements from Ross-Hellauer (2017): open identities or open reports; journals implementing only other OPR elements were excluded.
  • The final dataset updated Wolfram et al. (2019), expanding from 20 publishers and 174 OPR journals to 38 publishers and 617 OPR journals as of December 2019; data were stored in Excel and analyzed using cross-tabulations, queries, and qualitative assessment.

Limitations

  • The paper states there is no universal way to identify OPR journals, so the broad multi-source search was time-consuming and “hit-or-miss”; some OPR journals may have been missed if not indexed in searched databases or published by already-included publishers.
  • The paper states findings are a snapshot as of late 2019, the OPR landscape is changing quickly, and there may be regional or language bias due to lack of familiarity with publication languages.
  • The paper states publishers lack annual OPR metric data on optional open reports and open identities; the text also makes evident that reliance on crowdsourced or unverified sources such as Transpose and DOAJ required correction of misclassified, duplicated, or erroneous records.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11192-020-03488-4

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 4 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    500 of the 617 OPR journals (81.0%) are published by only five publishers: MDPI, SDI, BioMed Central, Frontiers Media S.A., and Kowsar.

    “500 of the OPR journals (81.0%) are published by only five publishers (MDPI, SDI, BioMed Central, Frontiers Media S.A., Kowsar).”
    Quote in the paperNot a substantial claim
  • Findingnone given

    Medical and Health Sciences had the most early adopters.

    “Medical and Health Sciences had the most early adopters.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The websites and editorial policies of OPR journals accurately and completely describe their OPR practices and adoption dates.
  • Findingown data

    Five publishers are responsible for 81% of the identified OPR journals.

    “five publishers are responsible for 81% of the identified OPR journals”
    Quote in the paperSubstantial claim
  • Findingnone given

    F1000Research makes the review process itself open in addition to open identities and open reports.

    “F1000Research, the first of its kind, makes the review process itself open in addition to open identities and open reports.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The vast majority of OPR journals (608 or 98.5%) either require reviewers to identify themselves (268 or 43.4%) or allow reviewers to choose whether to identify themselves (340 or 55.1%).

    “The vast majority of OPR journals (608 or 98.5%) either require reviewers to identify themselves (268 or 43.4%) or allow reviewers to choose whether to identify themselves (340 or 55.1%).”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 19
  • May combine two claims. 23

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Atmospheric Chemistry and Physics and European Cells & Materials each implemented a different OPR model, although both launched their first issues in 2001.

    “The journals Atmospheric Chemistry and Physics and European Cells & Materials each implemented a different OPR model, although both launched their first issues in 2001.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The websites and editorial policies of OPR journals accurately and completely describe their OPR practices and adoption dates.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    F1000Research, as post-publication OPR, has no acceptance or rejection decision as a result of peer review, but an article will not be indexed in bibliographic databases without passing a threshold within a defined timeframe (two approved or one approved plus two approved with reservations).

    “F1000Research, as post-publication OPR, has no acceptance or rejection decision to be made as a result of peer review, but an article will not be indexed in any bibliographic databases without passing the threshold within a defined timeframe consisting of two approved (✔✔) or one approved (✔) plus two approved with reservations (??).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Since 2001, there has been steady growth in the number of journals that have adopted OPR, most noticeably in Medical and Health Sciences, and more recently in Natural Sciences over the past 10 years.

    “Since then, there has been steady growth in the number of journals that have adopted OPR, most noticeably in the Medical and Health Sciences, and more recently, in the Natural Sciences over the past 10 years.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The percentage of mandatory open reports is highest in Medical and Health Sciences (64.0%) and second highest in Multidisciplinary (50.0%).

    “The percentage of mandatory open reports is highest in the Medical and Health Sciences (64.0%), and second highest in the Multidisciplinary category (50.0%).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In many cases, OPR adoption coincided with the first journal issue.

    “In many cases, OPR adoption coincided with the first journal issue.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Twenty-four of the 38 identified publishers (63.2%) are based in Europe and account for 445 of the 617 OPR titles (72.1%).

    “Twenty-four of the 38 (63.2%) identified publishers are based in Europe and account for 445 out of the 617 titles (72.1%).”
    • Also stated in: § Discussion · plain
      “More than 60% of the publishers in this study, who publish more than 70% of the OPR journals identified, are based in Europe”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Only 189 (30.6%) OPR journals require both open identities and open reports, a minority of identified OPR journals.

    “Only 189 (30.6%) journals require both open identities and open reports.”
    • Also stated in: § Conclusion · plain
      “only a minority of the OPR journals identified have adopted complete transparency.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The adoption of OPR has gone from 38 journals in 2001 to at least 617 journals by the end of 2019.

    “Since the BMJ experiments with open reviews more than 20 years ago, the adoption of OPR has gone from 38 journals in 2001 to at least 617 journals by the end of 2019.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    As of 23 November 2019, the Transpose database listed 294 OPR journals that adopted open identities and 232 OPR journals that publish open reports, many of which were misclassified.

    “As of 23 November 2019, the Transpose database listed 294 OPR journals that adopted open identities and 232 OPR journals that publish open reports, many of which were misclassified perhaps due to the crowdsourcing nature of the database and the record contributors' ability to distinguish OA from OPR.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    PeerJ provides optional open identities to referees and optional open reports to authors.

    “PeerJ provides optional open identities to referees and optional open reports to authors, representing a range of journals adopting this model”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Mandatory open reports are much lower for Humanities (14.3%) and Technology (5.7%), where optional open reports are more common.

    “Mandatory open reports are much lower for Humanities (14.3%) and Technology (5.7%), where optional open reports are more common.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Frontiers OPR journals require open identities but do not provide open reports.

    “Frontiers' OPR journals publish only referee identities alongside articles without open reports as an open identities-only model”
    • Also stated in: § Discussion · plain
      “Frontiers Media S.A. requires open identities but does not provide open reports for its OPR journals.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    OPR adoption has shown steady growth since 2001, when 38 journals initially adopted OPR, with more rapid growth since 2017.

    “there has been a steady growth in OPR adoption since 2001, when 38 journals initially adopted OPR, with more rapid growth since 2017”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingnone given

    Current bibliographic databases do not systematically index OPR journals, and OPR journals do not clearly state their policies on open identities and open reports.

    “Current bibliographic databases do not systematically index OPR journals, nor do the OPR journals clearly state their policies on open identities and open reports.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    There has been steady growth in the number of OPR journals over time, led by journals in Medical and Health Sciences and Natural Sciences, with much higher growth since 2017.

    “Figure 1 demonstrates that there has been steady growth in the number of OPR journals over time, led by journals in Medical and Health Sciences and the Natural Sciences, but with much higher growth since 2017.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingnone given

    Information contained in sources such as DOAJ and Transpose is limited and frequently incorrect.

    “Information contained in sources such as DOAJ and Transpose is limited and frequently incorrect.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingnone given

    Most publishers post annual reports with metric data including number of articles, citation counts, Journal Impact Factor, rejection rate, etc., but they lack annual OPR metric data on the number or percentage of articles with optional open reports and open identities.

    “Although most publishers post annual reports with metric data including the number of articles, citation counts, Journal Impact Factor, rejection rate, etc., they lack annual OPR metric data on the number or percentage of articles with optional open reports and open identities; both are essential metric data to document OPR adoption.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    BioMed Central and SDI require both open reports and open identities.

    “BioMed Central, as one of the earliest OPR journal publishers in this study, and SDI require both open reports and open identities.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    MDPI makes open reports and open identities optional for authors and reviewers, respectively.

    “MDPI makes open reports and open identities optional for authors and reviewers, respectively.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    Kowsar requires open reports but makes referee identities optional.

    “Kowsar requires open reports but makes referee identities optional.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    First OPR usage was confirmed by searching early issues of the journals to identify when OPR practices began.

    “First OPR usage was confirmed by searching early issues of the journals to identify when OPR practices began.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    Several publishers have served as pioneers and early promoters of OPR.

    “Several publishers have served as pioneers and early promoters of OPR.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Across variations in OPR implementations, open identities and open reports are two important factors that define the degree of transparency.

    “Across the variations in OPR implementations, two important factors define the degree of transparency: open identities and open reports.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    If none of the articles published review reports or reviewer identities as of December 2019, the journal was excluded.

    “If none of the articles published review reports or reviewer identities as of December 2019, the journal was excluded.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The websites and editorial policies of OPR journals accurately and completely describe their OPR practices and adoption dates.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    To identify which year the journal started or transitioned to OPR, the authors accessed issues of the journals to find open reports or open identities in the published articles.

    “To identify which year the journal started or transitioned to OPR, we accessed issues of the journals to find open reports or open identities in the published articles.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The websites and editorial policies of OPR journals accurately and completely describe their OPR practices and adoption dates.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning

    When and where open reports can be accessed are important factors indicating the OPR transparency level.

    “When and where open reports can be accessed are also important factors indicating the OPR transparency level.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Each OPR journal's website was visited to verify the data.

    “Therefore, each OPR journal's website was visited to verify the data.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The websites and editorial policies of OPR journals accurately and completely describe their OPR practices and adoption dates.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The Humanities have different scholarship cultures compared to Natural Sciences and have been slow in adopting open access overall.

    “The Humanities have different scholarship cultures as compared to the Natural Sciences and have been slow in adopting open access overall (Eve 2017;Gross and Ryan 2015).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Eve 2017 source

      Not checked in the free run.

    • cites Gross and Ryan 2015 source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    If a journal publisher clearly posted information about when OPR was adopted on their editorial or peer review policy page, the authors used that year.

    “If a journal publisher clearly posted information about when OPR was adopted on their editorial or peer review policy page, we used that year (e.g., Kowsar and Wiley).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The websites and editorial policies of OPR journals accurately and completely describe their OPR practices and adoption dates.

  • Classified as probably not a substantial claim.

    Interpretationown data

    The level of transparency can be characterized along a continuum.

    “The level of transparency can be characterized along a continuum.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Several implementations allow authors and/or reviewers to participate in open review decisions during the process, including mandated open reports with optional open identities (e.g., Kowsar), mandated open reports without open identities (e.g., Ledger), and optional open reports with optional open identities (e.g., PeerJ).

    “The next several implementations that allow authors and/or reviewers to participate in open review decisions during the process include: mandated open reports but optional open identities (e.g., Kowsar journals), mandated open reports without open identities (e.g., the journal Ledger), and optional open reports with optional open identities (e.g., PeerJ).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    There are multiple approaches to the adoption of OPR, indicating there is no consensus at present regarding best practices.

    “The fact that there are multiple approaches to the adoption of OPR indicates there is no consensus at present regarding best practices.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Open identities and open reports are considered core elements to recognize OPR journals.

    “Of the seven elements, open identities and open reports are considered core elements to recognize OPR journals.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The approach of casting a broad net using multiple sources to identify candidate OPR journals is time-consuming and often hit-or-miss.

    “Our approach was to cast a broad net using multiple sources to identify candidate OPR journals, which is time-consuming and often hit-ormiss.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    It is possible that the study missed OPR journals not indexed by the databases searched or by the publishers already in the dataset, despite expanding searches to OPR publishers.

    “It is possible that we have missed OPR journals that are not indexed by the databases searched or by the publishers already in our dataset despite the fact that we expanded our searches to the OPR publishers to ensure inclusion.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The findings represent a snapshot as of late 2019.

    “Similarly, given the growth in the number of OPR journals over the past couple of years, the findings presented here represent a snapshot as of late 2019.”
    Quote in the paperNot a substantial claim
36 of 36

  • The paper relies on this without support.

    The websites and editorial policies of OPR journals accurately and completely describe their OPR practices and adoption dates.

    “Therefore, each OPR journal's website was visited to verify the data.”
    “If a journal publisher clearly posted information about when OPR was adopted on their editorial or peer review policy page, we used that year (e.g., Kowsar and Wiley).”

  • The paper relies on this without support.

    The multi-source search strategies identified all or most OPR journals existing as of 2019.

    “Using various methods, we identified 617 OPR journals that published at least one article with open identities or open reports as of 2019”
    “To identify relevant OPR journals, we used multiple search strategies and tracked different sources.”

  • The paper relies on this without support.

    The total population of scholarly journals is large enough that 617 OPR journals constitute a very small percentage.

    “However, the number of OPR journals remains a very small percentage of scholarly journals, overall.”

  • The paper relies on this without support.

    The headquarters location of publishers is a valid indicator of the geographic region leading OPR adoption.

    “Twenty-four of the 38 (63.2%) identified publishers are based in Europe and account for 445 out of the 617 titles (72.1%).”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The number of publishers and journals applying Open Peer Review increased from 20 publishers and 174 journals in 2018 to 38 publishers and 617 journals in 2020.

    “20 publishers and 174 journals in 2018 compared to 38 publishers and 617 journals in 2020, according to [Wolfram et al., 2019] and [Wolfram et al., 2020]”

    From The Open Review-Based (ORB) dataset: Towards Automatic Assessment of Scientific Papers and Experiment Proposals in High-Energy Physics (2023) · cites this paper as [Wolfram et al., 2020]

    The Data verification and cleaning section reports 20 publishers and 174 OPR journals in a dataset collected in 2018, and 38 publishers and 617 OPR journals in the expanded dataset. It dates the latter count to December 2019, not 2020. (LLM, conf=0.90)

    Experimental — accuracy not yet measured

  • As of December 2019, a total of 617 journals have taken up the practice of open peer review.

    “As of December 2019, a total of 617 journals have taken up the practice of open peer review (Wolfram et al., 2020).”

    From MOPRD: A multidisciplinary open peer review dataset (2022) · cites this paper as Wolfram et al., 2020

  • Open review is an exception rather than a rule.

    “despite its growing popularity (Wolfram et al., 2020), open review is an exception rather than a rule.”

    From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Wolfram et al., 2020

The citation lists haven’t been fetched yet

References 35 · Cited by 167

Research works printed but not linked: 14

  1. Agha, R. (2017). Publishing peer review reports. Webinar: Transparency in Peer Review. https://researcheracademy.elsevier.com/navigating-peer-review/fundamentals-peer-review/transparency-peer-review.
  2. ASAPbio (2018). Transparency, recognition, and innovation in peer review in the life sciences (February 2018)/Peer review survey results. https://asapbio.org/peer-review/survey.
  3. Clobridge, A. (2016). Open peer review: the next wave in open knowledge? Online Searcher: Information Discovery, Technology, Strategies, 40(4), 60–62.
  4. EMBO Press. (2020). Transparent Peer Review. https://www.embopress.org/policies.
  5. Fennell, C., Corney, A., & Ash E. (2017). Transparency—the key to trust in peer review, Elsevier Connect,https://www.elsevier.com/connect/transparency-the-key-to-trust-in-peer-review.
  6. Graf, C. (2019) Why more journals are joining our transparent peer review pilot. Director, Research Integrity and Publishing Ethics at Wiley. September 20, 2019. https://www.wiley.com/network/researchers/latest-content/why-more-journals-are-joining-our-transparent-peer-review-pilot.
  7. Mehmani, B. (2016). Is open peer review the way forward? https://www.elsevier.com/reviewers-update/story/innovation-in-publishing/is-open-peer-review-the-way-forward.
  8. Mehmani, B., & van Rossum, J. (2015). Elsevier trials publishing peer review reports as articles. https://www.elsevier.com/reviewers-update/story/peer-review/elsevier-pilot-trials-publishing-peer-review-reports-as-articles.
  9. Moylan, E. (2019). How do researchers feel about open peer review? Wiley. September 17, 2019. https://www.wiley.com/network/researchers/submission-and-navigating-peer-review/how-do-researchers-feel-about-open-peer-review.
  10. Moylan, E. (2020). Progressing towards transparency—more journals join our transparent peer review pilot. Wiley. March 5, 2020. https://www.wiley.com/network/researchers/submission-and-navigating-peer-review/progressing-towards-transparency-more-journals-join-our-transparent-peer-review-pilot.
  11. Nature (2020). Nature will publish peer review reports as a trial, Editorial 05 February 2020. Nature, 578. https://www.nature.com/articles/d41586-020-00309-9.
  12. Wang, P., & Wolfram, D. (2016). The last frontier in open science: Will open peer review transform scientific and scholarly publishing? at the 2016 Annual Meeting of the Association for Information Science and Technology, October 14–18, 2016, Copenhagen, Denmark. [Panellists: Jason Hoyt, PeerJ; Ulrich Pöschl, Max Planck; Peter Ingwersen, Royal School of Denmark & Richard Smith, retired editor of The BMJ; discussant: Marcia Bates, University of California, Log Angeles].
  13. Wiley (2018). Bringing greater transparency to peer review: Wiley and Clarivate analytics partner to launch innovative open peer review, 13 September 2018, https://newsroom.wiley.com/press-release/bringing-greater-transparency-peer-review-wiley-and-clarivate-analytics-partner-launch.
  14. Wiley (2020). A list of participating OPR journals is referred at https://authorservices.wiley.com/Reviewers/journal-reviewers/what-is-peer-review/index.html (direct link: https://authorservices.wiley.com/asset/photos/reviewers.html/Journals%20Included%20in%20Transparent%20Peer%20Review%20Pilot.xlsx).

Authors
Wolfram, D., Wang, P., Hembree, A., & Park, H.
Affiliations
  • University of Wisconsin–Milwaukee (US)
  • University of Tennessee at Knoxville (US)
Corresponding author
  • Wolfram, Dietmar
  • Wang, Peiling
  • Hembree, Adam
  • Park, Hyoungjoo
ORCID
Venue
Scientometrics
Publisher
Springer Science and Business Media LLC
Published
2020-05-26
Volume
125
Issue
2
Pages
1033-1051
ISSN
  • 0138-9130 (ISSN-L)
  • 1588-2861
DOI
10.1007/s11192-020-03488-4
OpenAlex
W3029733514
Semantic Scholar
43ab7b95ffd83829e919898df5aba3a958b9e802
Cited by
167
References
35
Citations by year
2026: 22 · 2025: 32 · 2024: 30 · 2023: 28 · 2022: 25 · 2021: 24 · 2020: 5
FWCI
20.35
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientometrics (publishedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
Funders
University of Wisconsin-Milwaukee
Topics
  • Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
Keywords
open peer review, open science, peer review transparency, reviewer anonymity, publication ethics, open identity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review