The sociology of scientific validity: How professional networks shape judgement in peer review
Open accessIntegrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Empirical social studies of peer review demonstrate the important role of feedback variation from reviewers.
“While it might be intuitive to expect that a standardization of text in peer reviews could be useful, empirical social studies of peer review demonstrate the important role of feedback variation from reviewers (Teplitskiy et al., 2018;Lamont, 2009;2012;Longino, 1990;Sulik et al., 2023).”
From Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews (2024) · cites this paper as Teplitskiy et al., 2018
Cites
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225 · see citing works
The Burden of Knowledge and the “Death of the Renaissance Man”: Is Innovation Getting Harder?
JONES, B. F. (2009) · Review of Economic Studies · doi:10.1111/j.1467-937x.2008.00531.x · Cited by 1,058 · see citing works
Reliability in content analysis: some common misconceptions and recommendations.
Krippendorff, K. (2004) · Human Communication Research · doi:10.1093/hcr/30.3.411 · Cited by 1,027 · see citing works
Chance and Consensus in Peer Review
Cole, S., Cole, J. R., & Simon, G. A. (1981) · Science · doi:10.1126/science.7302566 · Cited by 623 · see citing works
Could a neuroscientist understand a microprocessor?
Jonas, E., & Paul Kording, K. (2016) · doi:10.1101/055624 · Cited by 329 · see citing works
Does Masking Author Identity Improve Peer Review Quality?: A Randomized Controlled Trial
Justice, A. C., Cho, M. K., Winker, M. A., Berlin, J. A., Rennie, D., & and the PEER Investigators (1998) · JAMA · doi:10.1001/jama.280.3.240 · Cited by 278 · see citing works
Testing for the Presence of Positive-Outcome Bias in Peer Review: A Randomized Controlled Trial
Emerson, G. B., Warme, W. J., Wolf, F. M., Heckman, J. D., Brand, R. A., & Leopold, S. S. (2010) · Archives of Internal Medicine · doi:10.1001/archinternmed.2010.406 · Cited by 245 · see citing works
Bornmann, L., Mutz, R., & Daniel, H.-D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0014331 · Cited by 231 · see citing works
The Effects of Blinding on Acceptance of Research Papers by Peer Review
Fisher, M. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03520020069019 · Cited by 164 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
The limitations to our understanding of peer review
Tennant, J. P., & Ross-Hellauer, T. (2020) · Research Integrity and Peer Review · doi:10.1186/s41073-020-00092-1 · Cited by 384 · see citing works
Pandemic publishing: Medical journals strongly speed up their publication process for COVID-19
Horbach, S. P. J. M. (2020) · Quantitative Science Studies · doi:10.1162/qss_a_00076 · Cited by 268 · see citing works
Liang, W., Izzo, Z., Zhang, Y., Lepp, H., Cao, H., Zhao, X., Chen, L., Ye, H., Liu, S., Huang, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.07183 · Cited by 73 · see citing works
“While it might be intuitive to expect that a standardization of text in peer reviews could be useful, empirical social studies of peer review demonstrate the important role of feedback variation from reviewers (Teplitskiy et al., 2018; Lamont, 2009; 2012; Longino, 1990; Sulik et al., 2023).”
Dondio, P., Casnici, N., Grimaldo, F., Gilbert, N., & Squazzoni, F. (2019) · Journal of Informetrics · doi:10.1016/j.joi.2019.03.018 · Cited by 48 · see citing works
Messing with Merton: The intersection between Open Science practices and Mertonian values
Hosseini, M., Senabre Hidalgo, E., Horbach, S. P. J. M., Guttinger, S., & Penders, B. (2022) · doi:10.31235/osf.io/9nyh8 · Cited by 30 · see citing works
The oracles of science: On grant peer review and competitive funding
Roumbanis, L. (2021) · Social Science Information · doi:10.1177/05390184211019241 · Cited by 15 · see citing works
“…may nonetheless offer an important source of intellectual exchange between researchers – an ongoing ‘organized skepticism’ is taking place, for example, during seminars, conferences and workshops, and in journal peer review (cf. Brezis and Birukou, 2020; Fyfe et al., 2020; Teplitskiy et al., 2018).”
BackgroundLLM hallucinations in the wild: Large-scale evidence from non-existent citations
Zhao, Z., Wang, Y., Stuart, T., De Vaan, M., Ginsparg, P., & Yin, Y. (2026) · arXiv · doi:10.48550/arxiv.2605.07723 · Cited by 3 · see citing works
“Are hallucinated citations confined to lower-tier journals [33]?”
- Authors
- Teplitskiy, M., Acuna, D., Elamrani-Raoult, A., Körding, K., & Evans, J.
- Affiliations
- Harvard University (US)
- Syracuse University (US)
- École Normale Supérieure - PSL (FR)
- University of Chicago (US)
- Corresponding author
- Teplitskiy, Misha
- Evans, James
- Venue
- Research Policy
- Publisher
- Elsevier BV
- Published
- 2018-11
- Volume
- 47
- Issue
- 9
- Pages
- 1825-1841
- ISSN
- 0048-7333 (ISSN-L)
- OpenAlex
- W2786497440
- Cited by
- 113
- References
- 9
- Citations by year
- 2026: 10 · 2025: 21 · 2024: 12 · 2023: 13 · 2022: 21 · 2021: 13 · 2020: 12 · 2019: 9 · 2018: 2
- FWCI
- 6.86
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Research Policy (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Keywords
- peer review, scientific validity, reviewer bias, nepotism, co-authorship networks, professional networks, schools of thought, PLOS ONE
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
