CoChat
Sign Up
Cited by 6Publishing and Scholarly CommunicationAcademic Publishing and Open AccessAcademic Writing and Publishing

The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.

The role of author identities in peer review

Open access
Shah, Nihar B.·2023·PLOS ONE
DOI 10.1371/journal.pone.0286206PMC PMC10284400
Open this paper in CoChat — free accountOpen PDF

Abstract

There is widespread debate on whether to anonymize author identities in peer review. The key argument for anonymization is to mitigate bias, whereas arguments against anonymization posit various uses of author identities in the review process. The Innovations in Theoretical Computer Science (ITCS) 2023 conference adopted a middle ground by initially anonymizing the author identities from reviewers, revealing them after the reviewer had submitted their initial reviews, and allowing the reviewer to change their review subsequently. We present an analysis of the reviews pertaining to the identification and use of author identities. Our key findings are: (I) A majority of reviewers self-report not knowing and being unable to guess the authors’ identities for the papers they were reviewing. (II) After the initial submission of reviews, 7.1% of reviews changed their overall merit score and 3.8% changed their self-reported reviewer expertise. (III) There is a very weak and statistically insignificant correlation of the rank of authors’ affiliations with the change in overall merit; there is a weak but statistically significant correlation with respect to change in reviewer expertise. We also conducted an anonymous survey to obtain opinions from reviewers and authors. The main findings from the 200 survey responses are: (i) A vast majority of participants favor anonymizing author identities in some form. (ii) The “middle-ground” initiative of ITCS 2023 was appreciated. (iii) Detecting conflicts of interest is a challenge that needs to be addressed if author identities are anonymized. Overall, these findings support anonymization of author identities in some form (e.g., as was done in ITCS 2023), as long as there is a robust and efficient way to check conflicts of interest.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0286206

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Conducting reviewer surveys is commonplace at AI conferences.

    “Conducting reviewer surveys is commonplace at AI conferences (Névéol et al., 2017; Rastogi et al., 2022; Shah, 2023).”

    From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Shah, 2023

    The Survey section reports an optional survey of participants, including reviewers, at ITCS 2023. The Introduction identifies ITCS as a theoretical computer science conference; the paper does not establish that reviewer surveys are commonplace at AI conferences. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 29

  1. Manzoor E, Shah NB. Uncovering Latent Biases in Text: Method and Application to Peer Review. In: AAAI conference on Artificial Intelligence; 2021. · DOI
  2. RM Blank, 1991, The effects of double-blind versus single-blind reviewing: Experimental evidence from the American Economic Review, The American Economic Review, 1041
  3. JS Ross, 2006, Effect of blinded peer review on abstract acceptance, JAMA, 295, 1675, 10.1001/jama.295.14.1675 · DOI · Paper page
  4. Stelmakh I, Shah N, Singh A. On Testing for Biases in Peer Review. In: NeurlPS; 2019.
  5. Haffar S, Bazerbachi F, Murad MH. Peer review bias: a critical review. In: Mayo Clinic Proceedings. vol. 94. Elsevier; 2019. p. 670-676. · DOI
  6. Ware M. Peer review: benefits, perceptions and alternatives. Publishing Research Consortium. 2008;4:1-20.
  7. A Mulligan, 2013, Peer review in a changing world: An international study measuring the attitudes of researchers, Journal of the Association for Information Science and Technology, 64, 132
  8. Shah N, Draper S. IEEE Information Theory Society Report on the 2021 Diversity and Inclusion Survey; 2022. https://www.itsoc.org/sites/default/files/2022-04/ITSoc_D_I_Survey_Analysis.pdf.
  9. S Madden, 2006, Impact of double-blind reviewing on SIGMOD publication rates, ACM SIGMOD Record, 35, 29, 10.1145/1147376.1147381 · DOI
  10. Jecmen S, Zhang H, Liu R, Fang F, Conitzer V, Shah NB. Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design. In: Conference on Human Computation and Crowdsourcing (HCOMP); 2022. · DOI
  11. Beygelzimer A, Fox E, d’Alché Buc F, Larochelle H. What we learned from NeurIPS 2019 data; 2019. Available from: https://neuripsconf.medium.com/what-we-learned-from-neurips-2019-data-111ab996462c.
  12. Rastogi C, Stelmakh I, Shen X, Meila M, Echenique F, Chawla S, et al. To ArXiv or not to ArXiv: A Study Quantifying Pros and Cons of Posting Preprints Online. arXiv preprint arXiv:220317259. 2022;.
  13. C Le Goues, 2018, Effectiveness of anonymization in double-blind review, Communications of the ACM, 61, 30, 10.1145/3208157 · DOI
  14. S Hill, 2003, The myth of the double-blind review? Author identification using only citations, SIGKDD Explorations, 5, 179, 10.1145/980972.981001 · DOI
  15. Caragea C, Uban A, Dinu LP. The myth of double-blind review revisited: ACL vs. EMNLP. In: EMNLP-IJCNLP; 2019. p. 2317-2327. · DOI
  16. Matsubara Y, Singh S. Citations Beyond Self Citations: Identifying Authors, Affiliations, and Nationalities in Scientific Papers. In: International Workshop on Mining Scientific Publications; 2020. p. 9-20.
  17. Hicks M. POPL 2012: FAQ concerning double-blind reviewing.; 2012. http://www.cs.umd.edu/~mwh/dbr-faq.html.
  18. Smirnova I, Romero DM, Teplitskiy M. The bias-reducing effect of voluntary anonymization of authors’ identities: Evidence from peer review; 2023. https://papers.ssrn.com/abstract=4190623.
  19. De Ranieri E, McGillivray B, Swaminathan S, Samarasinghe M, Gruen L; 2017. Peer Review Congress.
  20. M Obrecht, 2007, Examining the value added by committee discussion in the review of applications for research awards, Research Evaluation, 16, 79, 10.3152/095820207X223785 · DOI
  21. M Fogelholm, 2012, Panel discussion does not improve reliability of peer review for medical research grant proposals, Journal of clinical epidemiology, 65, 47, 10.1016/j.jclinepi.2011.05.001 · DOI · Paper page
  22. Lawrence N, Cortes C. The NIPS Experiment; 2014. http://inverseprobability.com/2014/12/16/the-nips-experiment.
  23. E Pier, 2017, Your comments are meaner than your score: score calibration talk influences intra-and inter-panel variability during scientific grant peer review, Research Evaluation, 26, 1, 10.1093/reseval/rvw025 · DOI
  24. Beygelzimer A, Dauphin Y, Liang P, Wortman Vaughan J. The NeurIPS 2021 Consistency Experiment; 2021. https://blog.neurips.cc/2021/12/08/the-neurips-2021-consistency-experiment/.
  25. Rastogi C, Stelmakh I, Beygelzimer A, Dauphin YN, Liang P, Vaughan JW, et al. How do Authors’ Perceptions of their Papers Compare with Co-authors’ Perceptions and Peer-review Decisions? arXiv:221112966. 2022;.
  26. L Fortnow, 2009, Viewpoint time for computer science to grow up, Communications of the ACM, 52, 33, 10.1145/1536616.1536631 · DOI
  27. B Parhami, 2016, Low acceptance rates of conference papers considered harmful, Computer, 49, 70, 10.1109/MC.2016.106 · DOI
  28. MY Vardi, 2017, Divination by program committee, Communications of the ACM, 60, 7, 10.1145/3122847 · DOI
  29. Teplitskiy M, Ranu H, Gray GS, Menietti M, Guinan E, Lakhani KR. Do Experts Listen to Other Experts?: Field Experimental Evidence from Scientific Peer Review; 2019.

Cited by

Sorted by citations

Metadata

Authors
Shah, N. B.
Affiliations
Carnegie Mellon University (US)
Corresponding author
Shah, Nihar B.
ORCID
Shah, Nihar B.
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2023-06-21
Volume
18
Issue
6
Pages
e0286206
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0286206
PMC
PMC10284400
OpenAlex
W4381489059
Cited by
6
References
9
Citations by year
2026: 2 · 2025: 1 · 2024: 2 · 2023: 1
FWCI
1.90
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Library and Information Sciences › Publishing and Scholarly Communication
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
  • Social Sciences › Library and Information Sciences › Academic Writing and Publishing
Keywords
peer review, reviewer bias, reviewer expertise, conflicts of interest, conference review process, author affiliation
MeSH terms
Attitude, Humans, Peer Review, Surveys and Questionnaires, Bias, Peer Review, Research, Self Report

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review