CoChat
Sign Up
PreprintCited by 1

To ArXiv or not to ArXiv: A Study Quantifying Pros and Cons of Posting Preprints Online

Rastogi, Charvi, Stelmakh, Ivan, Shen, Xinwei, Meila, Marina, Echenique, Federico, Chawla, Shuchi, Shah, Nihar B.·2022·arXiv
DOI 10.48550/arxiv.2203.17259arXiv 2203.17259
Open this paper in CoChat — free account

Abstract

Double-blind conferences have engaged in debates over whether to allow authors to post their papers online on arXiv or elsewhere during the review process. Independently, some authors of research papers face the dilemma of whether to put their papers on arXiv due to its pros and cons. We conduct a study to substantiate this debate and dilemma via quantitative measurements. Specifically, we conducted surveys of reviewers in two top-tier double-blind computer science conferences -- ICML 2021 (5361 submissions and 4699 reviewers) and EC 2021 (498 submissions and 190 reviewers). Our three main findings are as follows. First, more than a third of the reviewers self-report searching online for a paper they are assigned to review. Second, conference policies restricting authors from publicising their work on social media or posting preprints before the review process may have only limited effectiveness in maintaining anonymity. Third, outside the review process, we find that preprints from better-ranked institutions experience a very small increase in visibility compared to preprints from other institutions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2203.17259

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Conducting reviewer surveys is commonplace at AI conferences.

    “Conducting reviewer surveys is commonplace at AI conferences (Névéol et al., 2017; Rastogi et al., 2022; Shah, 2023).”

    From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Rastogi et al., 2022

  • As part of assessing the paper, reviewers often use search engines to find related literature-and might encounter public versions of the manuscript under review, which can inadvertently expose the authors' identities and affect the evaluation.

    “As part of assessing the paper, reviewers often use search engines to find related literature-and might encounter public versions of the manuscript under review, which can inadvertently expose the authors' identities and affect the evaluation (Rastogi et al., 2022).”

    From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Rastogi et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 41

  1. Valeria Aman. Is there any measurable benefit in publishing preprints in the arXiv section quantitative biology? arXiv preprint arXiv:1411.1955, 2014. · arXiv
  2. Alina Beygelzimer, Yann Dauphin, Percy Liang, and Jennifer Wortman Vaughan. The NeurIPS 2021 consistency experiment. https://blog.neurips.cc/2021/12/08/ the-neurips-2021-consistency-experiment/, 2021.
  3. Homanga Bharadhwaj, Dylan Turpin, Animesh Garg, and Ashton Anderson. De-anonymization of authors through arXiv submissions during double-blind review. arXiv preprint arXiv:2007.00177, 2020. · arXiv
  4. Rebecca M Blank. The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from The American Economic Review. American Economic Review, 81(5):1041-1067, December 1991.
  5. Amber E. Budden, Tom Tregenza, Lonnie W. Aarssen, Julia Koricheva, Roosa Leimu, and Christopher J. Lortie. Double-blind review favours increased representation of female authors. Trends in ecology & evolution, 23 1:4-6, 2008.
  6. L. Charlin and R. S. Zemel. The Toronto Paper Matching System: An automated paper-reviewer assignment system. In ICML Workshop on Peer Reviewing and Publishing Models, 2013.
  7. William S. Cleveland and Clive Loader. Smoothing by local regression: Principles and methods. In Wolfgang Härdle and Michael G. Schimek (eds.), Statistical Theory and Computational Aspects of Smoothing, pp. 10-49, Heidelberg, 1996. Physica-Verlag HD. ISBN 978-3-642-48425-4.
  8. Daniele Fanelli. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PloS one, 4(5):e5738, 2009.
  9. Sergey Feldman, Kyle Lo, and Waleed Ammar. Citation count analysis for papers with preprints. arXiv preprint arXiv:1805.05238, 2018. · arXiv
  10. Claire Le Goues, Yuriy Brun, Sven Apel, E. Berger, Sarfraz Khurshid, and Yannis Smaragdakis. Effectiveness of anonymization in double-blind review. Communications of the ACM, 61:30 -33, 2018.
  11. Sergei Ivanov. ICML 2020. Comprehensive analysis of authors, organi- zations, and countries., 2020. https://medium.com/criteo-engineering/ icml-2020-comprehensive-analysis-of-authors-organizations-and-countries-c4d1bb847fde [Last Accessed: 3/15/2022].
  12. Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar B. Shah, Vincent Conitzer, and Fei Fang. Mitigating manipulation in peer review via randomized reviewer assignments. In NeurIPS, 2020.
  13. Silvia Knobloch-Westerwick, Carroll J. Glynn, and Michael Huge. The Matilda effect in science com- munication: An experiment on gender bias in publication quality perceptions and collaboration in- terest. Science Communication, 35(5):603-625, 2013. doi: 10.1177/1075547012472684. URL https: //doi.org/10.1177/1075547012472684. · DOI · Paper page
  14. Ari Kobren, Barna Saha, and Andrew McCallum. Paper matching with local fairness constraints. In ACM KDD, 2019.
  15. N. Lawrence and C. Cortes. The NIPS Experiment. http://inverseprobability.com/2014/12/16/ the-nips-experiment, 2014. [Online; accessed 11-June-2018].
  16. Ann M. Link. US and Non-US Submissions: An Analysis of Reviewer Bias. JAMA, 280(3):246-247, 07 1998. ISSN 0098-7484. doi: 10.1001/jama.280.3.246. URL https://doi.org/10.1001/jama.280.3.246. · DOI
  17. Samuel Madden and David DeWitt. Impact of double-blind reviewing on SIGMOD publication rates. ACM SIGMOD Record, 35(2):29-32, 2006.
  18. Emaad Manzoor and Nihar B Shah. Uncovering latent biases in text: Method and application to peer review. In AAAI, 2021.
  19. Brian C Martinson, Melissa S Anderson, and Raymond De Vries. Scientists behaving badly. Nature, 435 (7043):737-738, 2005.
  20. Robert K. Merton. The Matthew effect in science. Science, 159(3810):56-63, 1968. doi: 10.1126/science. 159.3810.56. URL https://www.science.org/doi/abs/10.1126/science.159.3810. · DOI · Paper page
  21. Andrew Myers. Condorcet internet voting service. https://civs1.civs.us/, 2003. [Online; accessed March-2021]. · DOI · Paper page
  22. Syavash Nobarany, Kellogg S Booth, and Gary Hsieh. What motivates people to review articles? The case of the human-computer interaction community. Journal of the Association for Information Science and Technology, 67(6):1358-1371, 2016.
  23. Ritesh Noothigattu, Nihar Shah, and Ariel Procaccia. Loss functions, axioms, and peer review. Journal of Artificial Intelligence Research, 2021.
  24. Charvi Rastogi, Xiangchen Song, Zhijing Jin, Ivan Stelmakh, Hal Daumé III, Kun Zhang, and Nihar B Shah. A randomized controlled trial on anonymizing reviewers to each other in peer review discussions. arXiv preprint arXiv:2403.01015, 2024. · arXiv
  25. David B Resnik, Christina Gutierrez-Ford, and Shyamal Peddada. Perceptions of ethical problems with scientific journal peer review: An exploratory study. Science and engineering ethics, 14(3):305-310, 2008.
  26. Seán Roberts and Tessa Verhoef. Double-blind reviewing at EvoLang 11 reveals gender bias. Journal of Language Evolution, 1:163-167, 07 2016. doi: 10.1093/jole/lzw009. · DOI
  27. Margaret W. Rossiter. The Matilda effect in science. Social Studies of Science, 23(2):325-341, 1993. URL http://www.jstor.org/stable/285482.
  28. Nihar Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike Von Luxburg. Design and analysis of the NIPS 2016 review process. JMLR, 19(1):1913-1946, 2018.
  29. Nihar B Shah. An overview of challenges, experiments, and computational solutions in peer review. Com- munications of the ACM (to appear). Preprint available at http://bit.ly/PeerReviewOverview, July 2021.
  30. Flaminio Squazzoni and Gandelli Claudio. Saint Matthew strikes again. An agent-based model of peer review and the scientific community structure. Journal of Informetrics, 6:265-27, 04 2012. doi: 10.1016/j.joi. 2011.12.005,. · DOI
  31. Ivan Stelmakh, Charvi Rastogi, Nihar B Shah, Aarti Singh, and Hal Daumé III. A large scale randomized controlled trial on herding in peer-review discussions. arXiv preprint arXiv:2011.15083, 2020. · arXiv
  32. Ivan Stelmakh, Nihar Shah, and Aarti Singh. PeerReview4All: Fair and accurate reviewer assignment in peer review. JMLR, 2021a.
  33. Ivan Stelmakh, Nihar Shah, Aarti Singh, and Hal Daumé III. A novice-reviewer experiment to address scarcity of qualified reviewers in large conferences. In AAAI, 2021b.
  34. Ivan Stelmakh, Nihar Shah, Aarti Singh, and Hal Daumé III. Prior and prejudice: The novice reviewers' bias against resubmissions in conference peer review. In CSCW, 2021c.
  35. Ivan Stelmakh, Charvi Rastogi, Ryan Liu, Shuchi Chawla, Federico Echenique, and Nihar B Shah. Cite- seeing and reviewing: A study on citation bias in peer review. arXiv preprint arXiv:2203.17239, 2022. · arXiv
  36. Mengyi Sun, Jainabou Barry Danfa, and Misha Teplitskiy. Does double-blind peer review reduce bias? Evidence from a top computer science conference. Journal of the Association for Information Science and Technology, 2021.
  37. Warren Thorngate and Wahida Chowdhury. By the numbers: Track record, flawed reviews, journal space, and the fate of talented authors. Advances in Intelligent Systems and Computing, 229, 01 2013. doi: 10.1007/978-3-642-39829-2 · DOI
  38. Leanne Tite and Sara Schroter. Why do peer reviews decline to review? A survey. Journal of epidemiology and community health, 61:9-12, 02 2007. doi: 10.1136/jech.2006.049817. · DOI
  39. Andrew Tomkins, Min Zhang, and William Heavlin. Reviewer bias in single-versus double-blind peer review. Proceedings of the National Academy of Sciences, 114:201707323, 11 2017. doi: 10.1073/pnas.1707323114. · DOI · Paper page
  40. Anthony KH Tung. Impact of double blind reviewing on SIGMOD publication: A more detail analysis. ACM SIGMOD Record, 35(3):6-7, 2006.
  41. Boya Xie, Zhihong Shen, and Kuansan Wang. Is preprint the future of science? A thirty year journey of online preprint services. ArXiv, abs/2102.09066, 2021. · arXiv

Cited by

Sorted by citations

Authors
Rastogi, C., Stelmakh, I., Shen, X., Meila, M., Echenique, F., Chawla, S., & Shah, N. B.
Venue
arXiv
Publisher
arXiv
Published
2022
DOI
10.48550/arxiv.2203.17259
arXiv
2203.17259
References
1
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review