CoChat
Sign Up
Cited by 28Game Theory and Voting SystemsRecommender Systems and TechniquesAcademic Writing and Publishing

Loss Functions, Axioms, and Peer Review

Open access
Noothigattu, Ritesh, Shah, Nihar, Procaccia, Ariel·2021·Journal of Artificial Intelligence Research
DOI 10.1613/jair.1.12554
Open this paper in CoChat — free accountOpen PDF

Abstract

It is common to see a handful of reviewers reject a highly novel paper, because they view, say, extensive experiments as far more important than novelty, whereas the community as a whole would have embraced the paper. More generally, the disparate mapping of criteria scores to final recommendations by different reviewers is a major source of inconsistency in peer review. In this paper we present a framework inspired by empirical risk minimization (ERM) for learning the community's aggregate mapping. The key challenge that arises is the specification of a loss function for ERM. We consider the class of L(p,q) loss functions, which is a matrix-extension of the standard class of Lp losses on vectors; here the choice of the loss function amounts to choosing the hyperparameters p and q. To deal with the absence of ground truth in our problem, we instead draw on computational social choice to identify desirable values of the hyperparameters p and q. Specifically, we characterize p=q=1 as the only choice of these hyperparameters that satisfies three natural axiomatic properties. Finally, we implement and apply our approach to reviews from IJCAI 2017.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1613/jair.1.12554

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Higher loss in the aggregate function indicates more variability in how evaluators map criteria to overall scores, suggesting higher commensuration bias.

    “Higher loss indicates that there is more variability in how evaluators map criteria to overall scores, suggesting higher commensuration bias.”

    From Peer reviews of peer reviews: A randomized controlled trial and other experiments (2025) · cites this paper as [40]

    The Introduction (page not available) defines commensuration bias as reviewers differing in how they weight criteria, and Section 4.2 defines reviewer loss as the difference between an overall score and the aggregate mapping’s output. The paper does not establish that higher loss specifically measures greater variability in reviewers’ mappings or, therefore, higher commensuration bias. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Existing solutions to commensuration bias are limited due to their reliance on numeric scores alone.

    “Existing solutions to commensuration bias such as (Noothigattu et al., 2021) are limited due to their reliance on numeric scores alone, and NLP can substantially contribute to this line of research by helping extract individual preferences from reviewers' aggregate review reports.”

    From What Can Natural Language Processing Do for Peer Review? (2024) · cites this paper as Noothigattu et al., 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. Alon, N., Fischer, F., Procaccia, A., and Tennenholtz, M. (2011). Sum of us: Strategyproof selection from the selectors. In Proceedings of the 13th Conference on Theoretical Aspects of Rationality and Knowledge, pages 101-110. ACM.
  2. Arrow, K. (1951). Social Choice and Individual Values. Wiley.
  3. Bakanic, V., McPhail, C., and Simon, R. J. (1987). The manuscript review and decision- making process. American Sociological Review, 52(5):631-642.
  4. Balietti, S., Goldstone, R. L., and Helbing, D. (2016). Peer review and competition in the art exhibition game. Proceedings of the National Academy of Sciences, 113(30):8414-8419.
  5. Brandt, F., Conitzer, V., Endriss, U., Lang, J., and Procaccia, A. D., editors (2016). Handbook of Computational Social Choice. Cambridge University Press.
  6. Cai, T., Liu, W., and Luo, X. (2011). A constrained ℓ-1 minimization approach to sparse pre- cision matrix estimation. Journal of the American Statistical Association, 106(494):594- 607.
  7. Chatterjee, S., Guntuboyina, A., and Sen, B. (2018). On matrix estimation under mono- tonicity constraints. Bernoulli, 24(2):1072-1100.
  8. Church, K. (2005). Reviewing the reviewers. Computational Linguistics, 31(4):575-578.
  9. Ding, C., Zhou, D., He, X., and Zha, H. (2006). R 1 -PCA: Rotational invariant L 1 -norm principal component analysis for robust subspace factorization. In Proceedings of the International Conference on Machine Learning (ICML), pages 281-288.
  10. Ding, W., Shah, N. B., and Wang, W. (2020). On the privacy-utility tradeoff in peer-review data analysis. In AAAI Privacy-Preserving Artificial Intelligence (PPAI-21) workshop.
  11. Gao, F. and Wellner, J. A. (2007). Entropy estimate for high-dimensional monotonic func- tions. Journal of Multivariate Analysis, 98(9):1751-1764.
  12. Hojat, M., Gonnella, J. S., and Caelleigh, A. S. (2003). Impartial judgment by the "gate- keepers" of science: Fallibility and accountability in the peer review process. Advances in Health Sciences Education, 8(1):75-96.
  13. Jecmen, S., Zhang, H., Liu, R., Shah, N. B., Conitzer, V., and Fang, F. (2020). Mitigating manipulation in peer review via randomized reviewer assignments. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS).
  14. Kahng, A., Kotturi, Y., Kulkarni, C., Kurokawa, D., and Procaccia, A. (2018). Ranking wily people who rank each other. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
  15. Kashlak, A. B. and Kong, L. (2021). Nonasymptotic support recovery for high-dimensional sparse covariance matrices. Stat, 10(1):e316.
  16. Kerr, S., Tolliver, J., and Petree, D. (1977). Manuscript characteristics which influence acceptance for management and social science journals. Academy of Management Journal, 20(1):132-141.
  17. Kong, D., Ding, C., and Huang, H. (2011). Robust nonnegative matrix factorization using L21-norm. In Proceedings of the International Conference on Information and Knowledge Management (CIKM).
  18. Kowalski, M. (2009). Sparse regression using mixed norms. Applied and Computational Harmonic Analysis, 27(3):303-324.
  19. Kurokawa, D., Lev, O., Morgenstern, J., and Procaccia, A. D. (2015). Impartial peer review. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI).
  20. Lamont, M. (2009). How Professors Think. Harvard University Press.
  21. Lawrence, N. and Cortes, C. (2014). The NIPS Experiment. http://inverseprobability. com/2014/12/16/the-nips-experiment.
  22. Mahoney, M. J. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1(2):161-175.
  23. Manzoor, E. and Shah, N. B. (2021). Uncovering latent biases in text: Method and appli- cation to peer review. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
  24. Masnadi-Shirazi, H. and Vasconcelos, N. (2008). On the design of loss functions for classi- fication: Theory, robustness to outliers, and SavageBoost. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS).
  25. Mei, S., Bai, Y., and Montanari, A. (2018). The landscape of empirical risk for nonconvex losses. Annals of Statistics, 46(6A):2747-2774.
  26. Mogul, J. C. and Anderson, T. (2008). Before and after wowcs: A literature survey, a list of papers we wish had been submitted. In Workshop on Organizing Workshops, Conferences, and Symposia for Computer Systems (WOWCS).
  27. Nie, F., Huang, H., Cai, X., and Ding, C. H. (2010). Efficient and robust feature selection via joint ℓ 2,1 -norms minimization. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS).
  28. Nielsen, M. W., Baker, C. F., Brady, E., Petersen, M. B., and Andersen, J. P. (2021). Meta-research: Weak evidence of country-and institution-related status bias in the peer review of abstracts. Elife.
  29. Rahimpour, A., Taalimi, A., and Qi, H. (2017). Feature encoding in band-limited distributed surveillance systems. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE.
  30. Roos, M., Rothe, J., and Scheuermann, B. (2011). How to calibrate the scores of biased reviewers by quadratic programming. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
  31. Rosasco, L., Vito, E. D., Caponnetto, A., Piana, M., and Verri, A. (2004). Are loss functions all the same? Neural Computation, 16(5):1063-1076.
  32. Shah, N. B., Balakrishnan, S., Guntuboyina, A., and Wainwright, M. J. (2017). Stochas- tically transitive models for pairwise comparisons: Statistical and computational issues. IEEE Transactions on Information Theory, 63(2):934-959.
  33. Shah, N. B., Balakrishnan, S., and Wainwright, M. J. (2019). Low permutation-rank matri- ces: Structural properties and noisy completion. Journal of Machine Learning Research.
  34. Shah, N. B., Balakrishnan, S., and Wainwright, M. J. (2020). A permutation-based model for crowd labeling: Optimal estimation and robustness. IEEE Transactions on Informa- tion Theory.
  35. Shah, N. B., Tabibian, B., Muandet, K., Guyon, I., and Von Luxburg, U. (2018). Design and analysis of the NIPS 2016 review process. The Journal of Machine Learning Research, 19(1):1913-1946.
  36. Spain, P. G. (1996). The Fermat point of a triangle. Mathematics Magazine, 69(2):131-133.
  37. Stelmakh, I., Rastogi, C., Shah, N. B., Singh, A., and Daumé III, H. (2020). A large scale randomized controlled trial on herding in peer-review discussions. arXiv preprint arXiv:2011.15083. · arXiv
  38. Stelmakh, I., Shah, N., and Singh, A. (2019a). PeerReview4All: Fair and accurate reviewer assignment in peer review. In Proceedings of the Conference on Algorithmic Learning Theory.
  39. Stelmakh, I., Shah, N., and Singh, A. (2021a). Catch me if i can: Detecting strategic behaviour in peer assessment. In Proceedings of the AAAI Conference on Artificial In- telligence (AAAI).
  40. Stelmakh, I., Shah, N., Singh, A., and Daum III, H. (2021b). A novice-reviewer experiment to address scarcity of qualified reviewers in large conferences. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
  41. Stelmakh, I., Shah, N., Singh, A., and Daum III, H. (2021c). Prior and prejudice: The novice reviewers' bias against resubmissions in conference peer review. In Proceedings of the Conference on Computer Supported Cooperative Work and Social Computing (CSCW).
  42. Stelmakh, I., Shah, N. B., and Singh, A. (2019b). On testing for biases in peer review. In Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS).
  43. Vijaykumar, T. N. (2020a). Potential organized fraud in ACM/IEEE com- puter architecture conferences. Online https://medium.com/@tnvijayk/ potential-organized-fraud-in-acm-ieee-computer-architecture-conferences-ccd61169370d Last accessed April 4, 2021.
  44. Vijaykumar, T. N. (2020b). Potential organized fraud in on- going ASPLOS reviews. Online https://medium.com/@tnvijayk/ potential-organized-fraud-in-on-going-asplos-reviews-874ce14a3ebe Last accessed April 4, 2021.
  45. Wang, J. and Shah, N. B. (2019). Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings. In Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS).
  46. Xu, Y., Zhao, H., Shi, X., and Shah, N. (2019). On strategyproof conference review. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI).
  47. Zhaoshui, H. and Cichocki, A. (2008). Sparse representation of L-order Markov signals. In International Symposium on Nonlinear Theory and its Applications.

Cited by

Sorted by citations

Authors
Noothigattu, R., Shah, N., & Procaccia, A.
Affiliations
  • Carnegie Mellon University (US)
  • Harvard University (US)
ORCID
Venue
Journal of Artificial Intelligence Research
Publisher
AI Access Foundation
Published
2021-04-25
Volume
70
Pages
1481-1515
ISSN
1076-9757 (ISSN-L)
DOI
10.1613/jair.1.12554
OpenAlex
W3008478051
Cited by
28
References
4
Citations by year
2025: 4 · 2024: 3 · 2023: 3 · 2022: 4 · 2021: 7 · 2020: 7
FWCI
3.25
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Artificial Intelligence Research (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (1763734, 1525932, 1350598, 1714140, 1733556, 1755656)
  • Office of Naval Research (N00014-17-1-2428, N00014-17-1, N00014-16-1-3075, N00014, N00014-17)
Topics
  • Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Game Theory and Voting Systems
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Academic Writing and Publishing
Keywords
peer review, loss functions, empirical risk minimization, axiomatic properties, computational social choice, review recommendation, hyperparameter optimization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review