Challenges, experiments, and computational solutions in peer review
Open accessAbstract
Improving the peer review process in a scientific manner shows promise.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Human expert reviewers may fail to detect major errors in papers and still recommend acceptance for flawed papers.
“Controlled studies have shown that human expert reviewers may fail to detect major errors in papers and still recommend acceptance for flawed papers (Shah, 2022, Section 10.1).”
From FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers (2025) · cites this paper as Shah, 2022, Section 10.1
The “Confirmation bias” section (page not provided) reports a controlled study in which reviewers were less likely to catch mistakes when a manuscript’s results agreed with their views. It does not establish that the mistakes were major or that reviewers recommended accepting the flawed manuscripts. (LLM, conf=0.85)
Experimental — accuracy not yet measuredPeer review faces well-documented challenges including inconsistency, miscalibration, and bias that undermine its reliability.
“peer review faces well-documented challenges - including inconsistency, miscalibration, and bias - that undermine its reliability (Shah, 2022).”
From FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers (2025) · cites this paper as Shah, 2022
Peer review of papers is plagued with well-documented problems such as bias, subjectivity, fraud, miscalibration, and low effort.
“While the peer review of papers is widespread, it is plagued with well-documented problems like bias, subjectivity, fraud, miscalibration, and low effort, among others (see [1] for a survey).”
From Peer reviews of peer reviews: A randomized controlled trial and other experiments (2025) · cites this paper as [1]
Reviewer abuse includes downscoring others' work to reduce potential competition, establishing collusion rings, and other behaviors.
“This includes downscoring others' work to reduce potential competition, establishing collusion rings in which a group of reviewers conspire to pick each others' work for review and give it a favourable evaluation (Section 4.3.2), and other behaviors (Shah, 2022, Section 4).”
From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Shah, 2022
Several algorithms have been developed to address this issue but have not performed well in practice.
“Several algorithms have been developed to address this issue (Roos et al., 2012; Ge et al., 2013) but have not performed well in practice (see Shah, 2022, Section 5 for more details on this miscalibration).”
From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Shah, 2022
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Reviewer bias in single- versus double-blind peer review
Tomkins, A., Zhang, M., & Heavlin, W. D. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1707323114 · Cited by 605 · see citing works
Effects of Editorial Peer Review: A Systematic Review
Jefferson, T., Alderson, P., Wager, E., & Davidoff, F. (2002) · JAMA · doi:10.1001/jama.287.21.2784 · Cited by 347 · see citing works
Expertise modeling for matching papers with reviewers
Mimno, D., & McCallum, A. (2007) · Knowledge Discovery and Data Mining · doi:10.1145/1281192.1281247 · Cited by 316 · see citing works
The ups and downs of peer review
Benos, D. J., Bashari, E., Chaves, J. M., Gaggar, A., Kapoor, N., LaFrance, M. W., Mans, R., Mayhew, D. L., McGowan, S. L., Polter, A. M., et al. (2007) · AJP Advances in Physiology Education · doi:10.1152/advan.00104.2006 · Cited by 296 · see citing works
Primacy and Recency Effects on Clicking Behavior
Murphy, J., Hofacker, C. F., & Mizerski, R. (2006) · Journal of Computer-Mediated Communication · doi:10.1111/j.1083-6101.2006.00025.x · Cited by 157 · see citing works
Peer review: reform or revolution?
Smith, R. (1997) · BMJ · doi:10.1136/bmj.315.7111.759 · Cited by 114 · see citing works
Porter, B., & Souther, A. (2023) · Machine Learning Research · doi:10.11648/j.mlr · Cited by 105 · see citing works
PrefLib: A Library for Preferences http://www.preflib.org
Mattei, N., & Walsh, T. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-41575-3_20 · Cited by 104 · see citing works
Garg, N., Kavitha, T., Kumar, A., Mehlhorn, K., & Mestre, J. (2010) · Algorithmica · doi:10.1007/s00453-009-9386-0 · Cited by 87 · see citing works
Peer review and competition in the Art Exhibition Game
Balietti, S., Goldstone, R. L., & Helbing, D. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1603723113 · Cited by 71 · see citing works
Longitudinal Trends in the Performance of Scientific Peer Reviewers
Callaham, M. L., & McCulloch, C. E. (2010) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2010.07.027 · Cited by 63 · see citing works
Brenner, L. A., Griffin, D., & Koehler, D. J. (2005) · Organizational Behavior and Human Decision Processes · doi:10.1016/j.obhdp.2005.02.002 · Cited by 53 · see citing works
Paper Matching with Local Fairness Constraints
Kobren, A., Saha, B., & McCallum, A. (2019) · Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3292500.3330899 · Cited by 52 · see citing works
Novel tools to streamline the conference review process
Flach, P. A., Spiegler, S., Golénia, B., Price, S., Guiver, J., Herbrich, R., Graepel, T., & Zaki, M. J. (2010) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1809400.1809413 · Cited by 38 · see citing works
How to Calibrate the Scores of Biased Reviewers by Quadratic Programming
Roos, M., Rothe, J., & Scheuermann, B. (2011) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v25i1.7847 · Cited by 38 · see citing works
Collusion rings threaten the integrity of computer science research
Littman, M. L. (2021) · Communications of the ACM · doi:10.1145/3429776 · Cited by 29 · see citing works
Uncovering Latent Biases in Text: Method and Application to Peer Review
Manzoor, E., & Shah, N. B. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i6.16608 · Cited by 28 · see citing works
Self-citationUser rankings from comparisons: Learning permutations in high dimensions
Mitliagkas, I., Gopalan, A., Caramanis, C., & Vishwanath, S. (2011) · Allerton Conference on Communication, Control, and Computing · doi:10.1109/allerton.2011.6120296 · Cited by 15 · see citing works
Artificial Intelligence Research
(2020) · Communications in computer and information science · doi:10.1007/978-3-030-66151-9 · Cited by 12 · see citing works
Flagged first, then by citations
Research works printed but not linked: 30
- Alon, N., Fischer, F., Procaccia, A., and Tennenholtz, M. Sum of us: Strategy proof selection from the selectors. In Proceedings of Conf. on Theoretical Aspects of Rationality and Knowledge, (2011).
- Anderson, T. Conference reviewing considered harmful. ACM SIGOPS Operating Systems Rev., (2009).
- Brown, T. Peer review and the acceptance of new scientific ideas: Discussion paper from a working party on equipping the public with an understanding of peer peview: November 2002-May 2004. Sense About Science, (2004).
- Charlin, L. and Zemel, R. The Toronto Paper Matching System: An automated paper-reviewer assignment system. In Proceedings of ICML Workshop on Peer Reviewing and Publishing Models, (2013).
- Fiez, T., Shah, N., and Ratliff, L. A SUPER* algorithm to optimize paper bidding in peer review. In Proceedings of Conf. Uncertainty in Artificial Intelligence, (2020).
- Guo, L., Wu, J., Chang, W., Wu, J., and Li, J. K-loop free assignment in conference review systems. In Proceedings of ICNC, (2018).
- Jecmen, S., Zhang, H., Liu, R., Shah, N., Conitzer, V., and Fang, F. Mitigating manipulation in peer review via randomized reviewer assignments. NeurIPS, (2020).
- Lauer, M. Case study in review integrity: Asking for favorable treatment. NIH Extramural Nexus, (2020).
- Le Goues, C., Brun, Y., Apel, S., Berger, E., Khurshid, S., and Smaragdakis, Y. Effectiveness of anonymization in double-blind review. Commun. ACM, (2018).
- Lee, C. Commensuration bias in peer review. Philosophy of Science, (2015).
- Lee, C., Sugimoto, C., Zhang, G., and Cronin, B. Bias in peer review. J. Assoc. Information Science and Technology, (2013).
- Leyton-Brown, K. and Mausam. AAAI 2021- Introduction; https://bit.ly/3r2L3Rr; (min. 8 onward).
- Mahoney, M. Publication prejudices: an experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research (1977).
- Markwood, I., Shen, D., Liu, Y., and Lu, Z. Mirage: Content masking attack against information-based online services. In Proceedings of USENIX Security Symp., (2017).
- Neubig, G., Wieting, J., McCarthy, A., Stent, A., Schluter, N., and Cohn, T. ACL reviewer matching code; https://github.com/acl-org/reviewerpaper-matching.
- Pier, E., Raclaw, J., Kaatz, A., Brauer, M., Carnes, M., Nathan, M., and Ford, C. Your comments are meaner than your score: Score calibration talk influences intra-and inter-panel variability during scientific grant peer review. Research Evaluation (2017).
- Rennie, D. Let's make peer review scientific. Nature, (2016).
- Rodriguez, M., Bollen, J., and Van de Sompel, H. Mapping the bid behavior of conference referees. J. Informetrics (2007).
- Shah, N., Tabibian, B., Muandet, K., Guyon, I., and Von Luxburg, U. Design and analysis of the NIPS 2016 review process. JMLR, (2018).
- Siegelman, S. Assassins and zealots: Variations in peer review. Radiology, (1991).
- Soergel, D., Saunders, A., and McCallum, A. Open scholarship and peer review: A time for experimentation, (2013).
- Stelmakh, I., Shah, N., and Singh, A. PeerReview4All: Fair and accurate reviewer assignment in peer review. JMLR, (2021).
- Stelmakh, I., Shah, N., Singh, A., Daumé III, H., and Rastogi, C. Experiments with the ICML 2020 peer-review process, (2020); https://blog.ml.cmu. edu/2020/12/01/icml2020exp/
- Teplitskiy, M., Ranub, H., Grayb, G., Meniettid, M., Guinan, E., and Lakhani, K. Social influence among experts: Field experimental evidence from peer review, (2019).
- Vijaykumar, T. Potential organized fraud in ACM/IEEE computer architecture conferences, (2020); https:// bit.ly/3o2Zjb3.
- Wang, J. and Shah, N. Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings. In Proceedings of AAMAS, (2019).
- Ware, M. Publishing research consortium peer review survey 2015. Publishing Research Consortium, (2016).
- Wing, J. and Chi, E. Reviewing peer review. Communications of the ACM (2011).
- Wu, R., Guo, C., Wu, F., Kidambi, R., van der Maaten, L., and Weinberger, K. Making paper reviewing robust to bid manipulation attacks, (2021); arXiv:2102.06020. · arXiv
- Xu, Y., Zhao, H., Shi, X., and Shah, N. On strategyproof conference review. In Proceedings of IJCAI, (2019).
Cited by
Sorted by citations
Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.01783 · Cited by 243 · see citing works
Peer Review in the Age of Generative AI
Kankanhalli, A., & National University of Singapore (2024) · Journal of the Association for Information Systems · doi:10.17705/1jais.00865 · Cited by 47 · see citing works
Towards theorizing peer review
Hug, S. E. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00195 · Cited by 22 · see citing works
ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing
Liu, R., & Shah, N. B. (2023) · arXiv · doi:10.48550/arxiv.2306.00622 · Cited by 23 · see citing works
“Their success in general purpose tasks also raises the natural question: can they be used for reviewing scientific papers (or proposals)? Peer review is highly strained due to fast increasing numbers of submissions and overburdening of reviewers [McC06; Sha22].”
BackgroundSelf-citationBiases in scholarly recommender systems: impact, prevalence, and mitigation
Färber, M., Coutinho, M., & Yuan, S. (2023) · Scientometrics · doi:10.1007/s11192-023-04636-2 · Cited by 28 · see citing works
Defining quality in peer review reports: a scoping review
Sizo, A., Lino, A. D. P., Rocha, Á., & Reis, L. P. (2025) · Knowledge and Information Systems · doi:10.1007/s10115-025-02435-0 · Cited by 26 · see citing works
Wu, W., Zhang, C., & Zhao, Y. (2025) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.70005 · Cited by 26 · see citing works
Reimagining peer review as an expert elicitation process
Marcoci, A., Vercammen, A., Bush, M., Hamilton, D. G., Hanea, A. M., Hemming, V., Wintle, B. C., Burgman, M. A., & Fidler, F. M. (2022) · BMC Research Notes · doi:10.1186/s13104-022-06016-0 · Cited by 22 · see citing works
Cite-seeing and reviewing: A study on citation bias in peer review
Stelmakh, I., Rastogi, C., Liu, R., Chawla, S., Echenique, F., & Shah, N. B. (2023) · PLoS ONE · doi:10.1371/journal.pone.0283980 · Cited by 26 · see citing works
Self-citationPeer reviews of peer reviews: A randomized controlled trial and other experiments
Goldberg, A., Stelmakh, I., Cho, K., Oh, A., Agarwal, A., Belgrave, D., & Shah, N. B. (2025) · PLOS ONE · doi:10.1371/journal.pone.0320444 · Cited by 18 · see citing works
“While the peer review of papers is widespread, it is plagued with well-documented problems like bias, subjectivity, fraud, miscalibration, and low effort, among others (see [Sha22] for a survey).”
InfluentialSelf-citationEditable User Profiles for Controllable Text Recommendations
Mysore, S., Jasim, M., McCallum, A., & Zamani, H. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591677 · Cited by 16 · see citing works
Large language models in peer review: challenges and opportunities
Sun, Z. (2025) · Scientometrics · doi:10.1007/s11192-025-05440-w · Cited by 15 · see citing works
Carbonell Cortés, C., Parra-Rojas, C., Pérez-Lozano, A., Arcara, F., Vargas-Sánchez, S., Fernández-Montenegro, R., Casado-Marín, D., Rondelli, B., & López-Verdeguer, I. (2024) · Data & Policy · doi:10.1017/dap.2024.41 · Cited by 15 · see citing works
Detecting LLM-generated peer reviews
Rao, V. S., Kumar, A., Lakkaraju, H., & Shah, N. B. (2025) · PLOS One · doi:10.1371/journal.pone.0331871 · Cited by 14 · see citing works
Self-citationSun, Z. (2024) · Journal of Informetrics · doi:10.1016/j.joi.2024.101501 · Cited by 13 · see citing works
Manipulation and peer mechanisms: A survey
Olckers, M., & Walsh, T. (2024) · Artificial Intelligence · doi:10.1016/j.artint.2024.104196 · Cited by 12 · see citing works
A large-scale randomized study of large language model feedback in peer review
Thakkar, N., Yuksekgonul, M., Silberg, J., Garg, A., Peng, N., Sha, F., Yu, R., Vondrick, C., & Zou, J. (2026) · Nature Machine Intelligence · doi:10.1038/s42256-026-01188-x · Cited by 11 · see citing works
Past, Present, and Future of Citation Practices in HCI
Oppenlaender, J. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713556 · Cited by 13 · see citing works
Analyzing Cyber Security Research Practices through a Meta-Research Framework
Pochat, V. L., & Joosen, W. (2023) · Cyber Security Experimentation and Test Workshop (CSET) · doi:10.1145/3607505.3607523 · Cited by 9 · see citing works
Meta-HCI: First Workshop on Meta-Research in HCI
Oppenlaender, J., Malacria, S., Fang, X., van Berkel, N., Chevalier, F., Yatani, K., & Hosio, S. J. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3706723 · Cited by 10 · see citing works
A Dataset on Malicious Paper Bidding in Peer Review
Jecmen, S., Yoon, M., Conitzer, V., Shah, N. B., & Fang, F. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583424 · Cited by 8 · see citing works
Self-citationFairly Allocating Goods and (Terrible) Chores
Hosseini, H., Mammadov, A., & Wąs, T. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/305 · Cited by 7 · see citing works
Rastogi, C., Stelmakh, I., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J. W., Xue, Z., Daumé, H., Pierson, E., & Shah, N. B. (2024) · PLoS ONE · doi:10.1371/journal.pone.0300710 · Cited by 10 · see citing works
Self-citationIsotonic mechanism for exponential family estimation in machine learning peer review
Yan, Y., Su, W., & B, J. F. A. (2025) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1093/jrsssb/qkaf025 · Cited by 6 · see citing works
What Can Natural Language Processing Do for Peer Review?
Kuznetsov, I., Afzal, O. M., Dercksen, K., Dycke, N., Goldberg, A., Hope, T., Hovy, D., Kummerfeld, J. K., Lauscher, A., Leyton-Brown, K., et al. (2024) · arXiv · doi:10.48550/arxiv.2405.06563 · Cited by 6 · see citing works
Self-citationLin, T.-L., Chen, W.-C., Hsiao, T.-F., Liu, H.-I., Yeh, Y.-H., Chan, Y.-K., Lien, W.-S., Kuo, P.-Y., Yu, P. S., & Shuai, H.-H. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.259 · Cited by 6
FairColor: An efficient algorithm for the Balanced and Fair Reviewer Assignment Problem
Bouanane, K., Medakene, A. N., Benbelghit, A., & Belhouari, S. B. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103865 · Cited by 5 · see citing works
Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
Thakkar, N., Yuksekgonul, M., Silberg, J., Garg, A., Peng, N., Sha, F., Yu, R., Vondrick, C., & Zou, J. (2025) · arXiv · doi:10.48550/arxiv.2504.09737 · Cited by 4 · see citing works
“Other studies have shown the potential of LLMs to make the entire review pipeline more efficient across various stages [32, 33, 34, 35] such as writing manuscripts [36], initial quality control [37, 38, 27], and even providing AI-generated instructions for how to write reviews [39].”
Graham, C. L. B., Landrain, T. E., Vjestica, A., Masselot, C., Lawton, E., Blondel, L., Haenal, L., Tzovaras, B. G., & Santolini, M. (2023) · F1000Research · doi:10.12688/f1000research.125886.2 · Cited by 4 · see citing works
A Conceptual Model for Claim Validation Based on Cryptographically Signed Data
Abe, R., Suzuki, S., & Nakamura, O. (2024) · IEEE Access · doi:10.1109/access.2024.3524509 · Cited by 3 · see citing works
Huang, J., Huang, W.-B., Bu, Y., Cao, Q., Shen, H., & Cheng, X. (2023) · Journal of Informetrics · doi:10.1016/j.joi.2023.101427 · Cited by 4 · see citing works
Шиян, А. А., Нікіфорова, Л., & Zharinov, S. S. (2024) · Актуальні питання у сучасній науці · doi:10.52058/2786-6300-2024-6(24)-368-381 · Cited by 3 · see citing works
The Sharply Decreasing Disruptiveness of HCI
Chen, Z., & Li, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713917 · Cited by 3 · see citing works
Removing Bias and Incentivizing Precision in Peer-grading
Chakraborty, A., Jindal, J., & Nath, S. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15329 · Cited by 5 · see citing works
The role of author identities in peer review
Shah, N. B. (2023) · PLoS ONE · doi:10.1371/journal.pone.0286206 · Cited by 4 · see citing works
Self-citationInto the Unknown: Assigning Reviewers to Papers with Uncertain Affinities
Cousins, C., Payan, J., & Zick, Y. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43254-5_11 · Cited by 3 · see citing works
Computational Experiments in Computer Science Research: A Literature Survey
Landeta-López, P., & Guevara‐Vega, C. (2024) · IEEE Access · doi:10.1109/access.2024.3458808 · Cited by 3 · see citing works
Dershowitz, N., & Verma, R. M. (2023) · Communications of the ACM · doi:10.1145/3584664 · Cited by 3 · see citing works
An Isotonic Mechanism for Overlapping OwnershipPreprint
Guo, Y., Xu, H., Wu, J., & Su, W. (2023) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2306.11154 · Cited by 6 · see citing works
Co-Reviewer: can AI review like a human? An agentic framework for LLM-human alignment in peer review
Bharti, P. K., Dalal, V., & Panchal, M. (2026) · Scientometrics · doi:10.1007/s11192-026-05557-6 · Cited by 2 · see citing works
ChatGPT-4 frente a evaluación humana para la revisión por pares en cardiología
Fernández‐Cisnal, A., y, P. A., Rama, D. F., García‐Pavía, P., Sanchís, L., & Sanchís, J. M. (2024) · Revista Española de Cardiología · doi:10.1016/j.recesp.2024.02.018 · Cited by 2 · see citing works
Carpenter, J. M., Corvillón, A., & Shah, N. B. (2025) · Publications of the Astronomical Society of the Pacific · doi:10.1088/1538-3873/adb5c1 · Cited by 2 · see citing works
Self-citationHorváth, I., Wan, T. T. H., Huang, J., Coatanéa, É., Rayz, J. M., Zeng, Y., & Kim, K.-Y. J. (2023) · Journal of Integrated Design and Process Science · doi:10.3233/jid-230046 · Cited by 2 · see citing works
Eliciting Honest Information from Authors Using Sequential Review
Zhang, Y., Schoenebeck, G. R., & Su, W. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28860 · Cited by 2 · see citing works
A FAIR and Free Prompt-Based Research Assistant
Shamsabadi, M., & D’Souza, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70242-6_21 · Cited by 2 · see citing works
A randomized controlled trial on anonymizing reviewers to each other in peer review discussions
Rastogi, C., Song, X., Jin, Z., Stelmakh, I., Daumé, H., Zhang, K., & Shah, N. B. (2024) · PLoS ONE · doi:10.1371/journal.pone.0315674 · Cited by 3 · see citing works
Self-citationIdentifying the Best Agent in a NetworkPreprint
Baumann, L. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2873888 · Cited by 2 · see citing works
Robust implementation with peer mechanisms and evidence
Baumann, L. (2025) · Games and Economic Behavior · doi:10.1016/j.geb.2025.10.007 · Cited by 1 · see citing works
Bharti, P. K., Panchal, M., & Dalal, V. (2026) · International Journal of Data Science and Analytics · doi:10.1007/s41060-026-01047-3 · Cited by 1 · see citing works
Peer review in a pickle: Policy approaches for academic peer review
Byers, B. S., Dougherty, T. R., & Horo, U. (2023) · MIT Science Policy Review · doi:10.38105/spr.dx30fr4j7c · Cited by 1 · see citing works
Optimal Allocation with Peer InformationPreprint
Niemeyer, A., & Preusser, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4952502 · Cited by 1 · see citing works
FLAWS: A Benchmark for Error Identification and Localization in Scientific Papers
Xi, S., Rao, V., Payan, J., & Shah, N. B. (2025) · arXiv · doi:10.48550/arxiv.2511.21843
“Controlled studies have shown that human expert reviewers may fail to detect major errors in papers and still recommend acceptance for flawed papers (Shah, 2022, Section 10.1).”
InfluentialSelf-citationUse and Effects of LLMs in Peer Review: A Randomized Experiment and Survey at ICML 2026
Kim, S. S. Y., Deng, W. H., Vaughan, J. W., Su, B., Su, W., Agarwal, A., Li, S., Jaggi, M., Goldstein, D. G., Shah, N. B., et al. (2026) · arXiv · doi:10.48550/arxiv.2609.19420
Self-citationDoes Appearance Influence Homework Rating?
Wu, J., Dang, X., Shi, J., & Jiang, W. (2022) · International Conference on Virtual Reality, Human-Computer Interaction and Artificial Intelligence (VRHCIAI) · doi:10.1109/vrhciai57205.2022.00044
A Bilevel Formalism for the Peer-Reviewing Problem
Auricchio, G., Zhang, R., Zhang, J., & Cai, X. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230263
ChatGPT-4 versus human assessment in cardiology peer review
Fernández‐Cisnal, A., y, P. A., Rama, D. F., García‐Pavía, P., Sanchís, L., & Sanchís, J. M. (2024) · Revista Española de Cardiología (English Edition) · doi:10.1016/j.rec.2024.02.004
Automation of Scientific Reviewer Assignment: a Survey of Problems and Methods
Petrychko, M., & Shtovba, S. (2024) · Visnyk of Vinnytsia Politechnical Institute · doi:10.31649/1997-9266-2024-172-1-56-64
On Truthful Item-Acquiring Mechanisms for Reward Maximization
Shan, L., Zhang, S., Zhang, J., & Wang, Z. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645345
Impartial Peer Selection: An Annotated Reading List
Lev, O., Lyon, H., & Mattei, N. (2024) · ACM SIGecom Exchanges · doi:10.1145/3699824.3699834
Peer Review Market Design: Effort-Based Matching and Admission ControlPreprint
Fernandes, C., Siderius, J., & Singal, R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5205896
Dynamic optimization of peer review length using information density analysis
Bharti, P. K., & Bhathawala, T. (2025) · Scientometrics · doi:10.1007/s11192-025-05416-w
Identity Theft in AI Conference Peer Review
Shah, N. B., Bok, M., Liu, X., & McCallum, A. (2025) · Communications of the ACM · doi:10.1145/3736703
Self-citationBoudreau, K. J. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6025114
SAFE: Probabilistic Framework to Characterize Honest Reviewers
Kwatra, D., & Bhatnagar, V. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-23241-0_22
Unpacking the predictive power of peer review under scientific disruption: Evidence from AI research
Zhang, Y.-J., Jiang, Y.-F., Yan, P.-W., Jiang, Z.-R., Lin, C.-X., He, G., & Liu, X.-Z. (2026) · J. Informetrics · doi:10.1016/j.joi.2026.101811
Open science, closed peer review?
Charness, G. B., Dreber, A., Evans, D., Gill, A., & Toussaert, S. (2026) · Journal of the Economic Science Association · doi:10.1017/esa.2026.10030
Bae, A. J. (2026) · Transportation Research Interdisciplinary Perspectives · doi:10.1016/j.trip.2026.102058
Distributed Peer Review at ALMA: An Empirical Comparison with Panel-Based Review
Carpenter, J. M., & Corvillón, A. (2026) · Publications of the Astronomical Society of the Pacific · doi:10.1088/1538-3873/ae7f51
Butters, A., Marshall, M. B., Pinfield, S., Stafford, T., Bondarenko, A., Neubauer, B., Nuske, R., Schwidlinski, P., & Denecke, H. (2026) · Research Integrity and Peer Review · doi:10.1186/s41073-026-00246-7
Bogner, J., & Verdecchia, R. (2026) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/3820786.3820789
Trong, T., Huynh, B., Huỳnh, N. N. U., Nguyen, T. T., Nguyen, T., Nguyễn, M., Tran, T.-D., & Pham, V.-H. (2026) · International Conference on Multimedia Analysis and Pattern Recognition · doi:10.1109/mapr72750.2026.11685704
AI-assisted peer review: efficiency for reviewers, burden for authors—a narrative review
Babker, A. M. A. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1968385
Stop Automating Peer Review Without Rigorous Evaluation
Baumann, J., Pei, J., Koyejo, S., & Hovy, D. (2026) · arXiv · doi:10.48550/arxiv.2605.03202
“Submission volumes at major AI conferences have grown significantly in recent years (Yang et al., 2026), making it increasingly difficult to find a large enough pool of qualified reviewers (Aczel et al., 2021; Shah, 2022).”
- Authors
- Shah, N. B.
- Affiliations
- Carnegie Mellon University (US)
- Corresponding author
- Shah, Nihar B.
- ORCID
- Shah, Nihar B.
- Venue
- Communications of the ACM
- Published
- 2022-05-20
- Volume
- 65
- Issue
- 6
- Pages
- 76-87
- ISSN
- 0001-0782 (ISSN-L)
- 1557-7317
- OpenAlex
- W4281392565
- Cited by
- 66
- References
- 19
- Citations by year
- 2026: 13 · 2025: 14 · 2024: 20 · 2023: 15 · 2022: 3 · 2016: 1
- FWCI
- 56.23
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Communications of the ACM (publishedVersion, cc-by-sa)
- Funders
- National Science Foundation (CAREER 1942124)
- Topics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Publishing and Scholarly Communication
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- peer review, scholarly publishing, scholarly communication, research evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
