Has the Machine Learning Review Process Become More Arbitrary as the Field Has Grown? The NeurIPS 2021 Consistency Experiment
Open accessAbstract
We present the NeurIPS 2021 consistency experiment, a larger-scale variant of the 2014 NeurIPS experiment in which 10% of conference submissions were reviewed by two independent committees to quantify the randomness in the review process. We observe that the two committees disagree on their accept/reject recommendations for 23% of the papers and that, consistent with the results from 2014, approximately half of the list of accepted papers would change if the review process were randomly rerun. Our analysis suggests that making the conference more selective would increase the arbitrariness of the process. Taken together with previous research, our results highlight the inherent difficulty of objectively measuring the quality of research, and suggest that authors should not be excessively discouraged by rejected work.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The NeurIPS 2014 and 2021 consistency experiments found that around half of the accepted papers would have been rejected if the review process had been independently rerun, and that approximately 26% (in 2014) and 23% (in 2021) of papers received inconsistent accept-or-reject decisions across the two committees.
“the NeurIPS 2014 and 2021 consistency experiments, in which roughly 10% of submissions were independently reviewed by two committees, found that around half of the accepted papers would have been rejected if the review process had been independently rerun, and that approximately 26% (in 2014) and 23% (in 2021) of papers received inconsistent accept-or-reject decisions across the two committees”
From On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists (2026) · cites this paper as Beygelzimer et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Cicchetti, D. V. (1991) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00065675 · Cited by 496 · cited 1× · see citing works
“However, like other human judgments [8], reviews are noisy, and studies across fields have shown that reviewers often disagree with one another [4, 5, 7, 12, 13].”
Noise: A Flaw in Human Judgment by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein
Aldous, D. (2022) · The Mathematical intelligencer · doi:10.1007/s00283-022-10207-9 · Cited by 480
Bornmann, L., Mutz, R., & Daniel, H.-D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0014331 · Cited by 231 · cited 1× · see citing works
“However, like other human judgments [8], reviews are noisy, and studies across fields have shown that reviewers often disagree with one another [4, 5, 7, 12, 13].”
Panel discussion does not improve reliability of peer review for medical research grant proposals.
Fogelholm, M., Leppinen, S., Auvinen, A., Raitanen, J., Nuutinen, A., & Väänänen, K. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.05.001 · Cited by 83 · see citing works
Examining the value added by committee discussion in the review of applications for research awards
Obrecht, M., Tibelius, K., & Daloisio, G. (2007) · Research Evaluation · doi:10.3152/095820207x223785 · Cited by 52 · cited 1× · see citing works
“However, like other human judgments [8], reviews are noisy, and studies across fields have shown that reviewers often disagree with one another [4, 5, 7, 12, 13].”
Pier, E. L., Raclaw, J., Kaatz, A., Brauer, M., Carnes, M., Nathan, M. J., & Ford, C. E. (2017) · Research Evaluation · doi:10.1093/reseval/rvw025 · Cited by 48 · cited 1× · see citing works
“However, like other human judgments [8], reviews are noisy, and studies across fields have shown that reviewers often disagree with one another [4, 5, 7, 12, 13].”
Daniel Kahneman, Olivier Sibony and Cass R. Sunstein: Noise: a flaw in human judgment
Kay, J. (2022) · Business Economics · doi:10.1057/s11369-022-00251-w · Cited by 44 · cited 3×
“Subjectivity and noise are common in other forms of decision making as well [8] and while it may be possible to implement interventions to reduce the level of noise, we will never be able to avoid some level of randomness in the process.”
Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment
Cortes, C., & Lawrence, N. D. (2021) · arXiv · doi:10.48550/arxiv.2109.09774 · Cited by 15 · see citing works
Rastogi, C., Stelmakh, I., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J. W., Xue, Z., Daumé, H., Pierson, E., & Shah, N. B. (2024) · PLoS ONE · doi:10.1371/journal.pone.0300710 · Cited by 10 · see citing works
Self-citation
Flagged first, then by citations
6 of the 15 references couldn’t be matched to a known paper.
Research works printed but not linked: 7
- Samy Bengio, Inioluwa Deborah Raji, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan. A retrospective on the NeurIPS 2021 ethics re- view process. NeurIPS blog post, available at https://blog.neurips.cc/2021/12/03/ a-retrospective-on-the-neurips-2021-ethics-review-process/, 2021.
- Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan. Introducing the NeurIPS 2021 paper checklist. NeurIPS blog post, available at https://blog.neurips.cc/ 2021/03/26/introducing-the-neurips-2021-paper-checklist/, 2021.
- Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan. The NeurIPS 2021 consistency experiment. NeurIPS blog post, available at https://blog.neurips.cc/2021/ 12/08/the-neurips-2021-consistency-experiment/, 2021.
- John Langford. The NIPS experiment. CACM blog post, available at https://cacm.acm.org/ blogs/blog-cacm/181996-the-nips-experiment/fulltext, 2015.
- Neil D. Lawrence. The NIPS Experiment. http://inverseprobability.com/2014/12/16/ the-nips-experiment, 2014.
- Neil D. Lawrence. Peer review and the NIPS experiment. Transcript of talk available at http://inverseprobability.com/talks/lawrence-peer15/ peer-review-and-the-nips-experiment.html, 2015.
- Eric Price. The NIPS experiment. Blog post available at http://blog.mrtz.org/2014/12/15/ the-nips-experiment.html, 2014.
Cited by
Sorted by citations
The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review
Su, B., Zhang, J., Collina, N., Yan, Y., Li, D., Cho, K., Fan, J., Roth, A., & Su, W. (2025) · Journal of the American Statistical Association · doi:10.1080/01621459.2025.2510006 · Cited by 6 · see citing works
“For example, a randomized experiment at NeurIPS 2021—one of the three premier conferences in ML/AI, alongside ICML and ICLR—revealed that about half of the accepted papers would have been rejected upon a second round of reviews (Cortes and Lawrence 2021; Beygelzimer et al. 2023).”
Kim, S., Yoon, D., Gashteovski, K., Suk, J., Baek, J., Aggarwal, P., Wu, I., Zaverkin, V., Petkoski, S., Schrider, D. R., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.20668 · Cited by 1 · see citing works
Use and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026
Kim, S. S. Y., Deng, W. H., Vaughan, J. W., Su, B., Su, W., Agarwal, A., Li, S., Jaggi, M., Goldstein, D. G., Shah, N. B., et al. (2026) · arXiv · doi:10.48550/arxiv.2609.19420
“Computer Science (CS) conferences, in turn, have begun to publish explicit peer-review LLM policies and explore the use of LLM assistance in the review workflow [1, 2, 7, 22, 33–36, 39, 59, 83].”
Self-citationPRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers
Loc, N. P. P., La Viet, T. H., Khanh, T. T., Nguyen, D. A., Pham, T. A. N., Nguyen, T., Chawla, N. V., Buntine, W., Wong, K.-S., Doan, K. D., et al. (2026) · arXiv · doi:10.48550/arxiv.2605.26730
When authors are the best judges of their work
AlShebli, B. (2026) · Nature Computational Science · doi:10.1038/s43588-026-01046-1
- Authors
- Beygelzimer, A., Dauphin, Y. N., Liang, P., & Vaughan, J. W.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2306.03262
- OpenAlex
- W4379919435
- Cited by
- 2
- References
- 15
- Citations by year
- 2026: 1 · 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- peer review, NeurIPS, research quality assessment, machine learning conferences
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
