Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 4

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings

Wang, Jingyan, Shah, Nihar B.·2019·International Joint Conference on Autonomous Agents and Multiagent Systems
DOI 10.65109/gwnx2060
Open this paper in CoChat — free account

Abstract

A key step in building multi-agent systems is to gather data reported by the agents (people), in either cardinal (numeric ratings) or ordinal (rankings) form. Cardinal scores collected from people are well known to suffer from miscalibrations. A popular approach to address this issue is to assume simplistic models of miscalibration (such as linear biases) to de-bias the scores. This approach, however, often fares poorly because people's miscalibrations are typically far more complex and not well understood. It is widely believed that in the absence of simplifying assumptions on the miscalibration, the only useful information in practice from the cardinal scores is the induced ranking. In this paper we address the fundamental question of whether this widespread folklore belief is actually true. We consider cardinal scores with arbitrary (or even adversarially chosen) miscalibrations that is only required to be consistent with the induced ranking. We design rating-based estimators and prove that despite making no assumptions on the ratings, they strictly and uniformly outperform all possible estimators that rely on only the ranking. These estimators can be used as a plug-in to show the superiority of cardinal scores over ordinal rankings for a variety of applications, and we provide examples for A/B testing and ranking as a proof of concept. Our results thus provide novel fundamental insights in the eternal debate between cardinal and ordinal data.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.65109/gwnx2060

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 1 references

Research works printed but not linked: 45

  1. Ammar Ammar and Devavrat Shah. Efficient rank aggregation using partial data. In SIGMET- RICS/PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems, 2012.
  2. A. J. Baranchik. A family of minimax estimators of the mean of a multivariate normal distribution. Ann. Math. Statist., 41(2):642-645, 1970.
  3. Jacob P. Baskin and Shriram Krishnamurthi. Preference aggregation in group recommender systems for committee decision-making. In ACM Conference on Recommender Systems, RecSys, 2009.
  4. Yukino Baba and Hisashi Kashima. Statistical quality estimation for general crowdsourcing tasks. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2013.
  5. M. E. Bock. Minimax estimators of the mean of a multivariate normal distribution. Ann. Statist., 3(1):209-218, 1975.
  6. Wade D. Cook, Boaz Golany, Michal Penn, and Tal Raviv. Creating a consensus ranking of proposals from reviewers' partial ordinal rankings. Computers & Operations Research, 34(4):954-965, 2007.
  7. Thomas M. Cover. Pick the Largest Number, pages 152-152. Springer New York, New York, NY, 1987.
  8. Cynthia Dwork, Ravi Kumar, Moni Naor, and D. Sivakumar. Rank aggregation methods for the web. In International Conference on World Wide Web, 2001.
  9. John R Douceur. Paper rating vs. paper ranking. ACM SIGOPS Operating Systems Review, 43(2):117-121, 2009.
  10. Sanjoy Dasgupta, Christos H. Papadimitriou, and Umesh Vazirani. Algorithms. McGraw-Hill, Inc., 1 edition, 2008.
  11. Yoav Freund, Raj D. Iyer, Robert E. Schapire, and Yoram Singer. An efficient boosting algorithm for combining preferences. Journal of Machine Learning Research, 4:933-969, 2003.
  12. Peter A. Flach, Sebastian Spiegler, Bruno Golénia, Simon Price, John Guiver, Ralf Herbrich, Thore Graepel, and Mohammed J. Zaki. Novel tools to streamline the conference review process: Experiences from SIGKDD'09. SIGKDD Explor. Newsl., 11(2):63-67, 2010.
  13. Dale Griffin and Lyle Brenner. Perspectives on Probability Judgment Calibration, chapter 9, pages 177-199. Wiley-Blackwell, 2008.
  14. Alexander V. Gnedin and Ulrich Krengel. Optimal selection problems based on exchangeable trials. Ann. Appl. Probab., 6(3):862-882, 08 1996.
  15. Alexander V. Gnedin. A solution to the game of googol. Ann. Probab., 22(3):1588-1595, 07 1994.
  16. Alexander Gnedin. Guess the larger number. preprint arXiv:1608.01899, 2016. · arXiv
  17. Hong Ge, Max Welling, and Zoubin Ghahramani. A Bayesian model for calibrating conference review scores, 2013. http://mlg.eng.cam.ac.uk/hong/nipsrevcal.pdf [Online; accessed 14-May-2018].
  18. Anne-Wil Harzing, Joyce Baldueza, Wilhelm Barner-Rasmussen, Cordula Barzantny, Anne Canabal, Anabella Davila, Alvaro Espejo, Rita Ferreira, Axele Giroud, Kathrin Koester, et al. Rating versus ranking: What is the best way to reduce response and language bias in cross-national research? International Business Review, 18(4):417-432, 2009.
  19. Reinhard Heckel, Nihar B. Shah, Kannan Ramchandran, and Martin J. Wainwright. Active ranking from pairwise comparisons and when parametric assumptions don't help. preprint arXiv:1606.08842, 2016. · arXiv
  20. William James and Charles Stein. Estimation with quadratic loss. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1, pages 361-379, 1961.
  21. Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne. Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery, 18(1):140-181, 2009.
  22. John Langford. ICML acceptance statistics, 2012. http://hunch.net/?p=2517 [Online; accessed 14-May- 2018].
  23. Mark D. McDonnell and Derek Abbott. Randomized switching in the two-envelope problem. Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, 2009.
  24. Ioannis Mitliagkas, Aditya Gopalan, Constantine Caramanis, and Sriram Vishwanath. User rankings from comparisons: Learning permutations in high dimensions. In Allerton Conference on Communication, Control, and Computing, 2011.
  25. R. S. MacKay, R. Kenna, R. J. Low, and S. Parker. Calibration with confidence: a principled method for panel assessment. Royal Society Open Science, 4(2), 2017.
  26. Sahand Negahban, Sewoong Oh, and Devavrat Shah. Iterative ranking from pair-wise comparisons. In Advances in Neural Information Processing Systems, 2012.
  27. Ritesh Noothigattu, Nihar Shah, and Ariel Procaccia. Choosing how to choose papers. arXiv preprint arxiv:1808.09057, 2018. · arXiv
  28. S. R. Paul. Bayesian methods for calibration of examiners. British Journal of Mathematical and Statistical Psychology, 34(2):213-223, 1981.
  29. Arun Rajkumar, Suprovat Ghoshal, Lek-Heng Lim, and Shivani Agarwal. Ranking from stochastic pairwise preferences: Recovering Condorcet winners and tournament solution sets at the top. In International Conference on Machine Learning, pages 665-673, 2015.
  30. Herbert Robbins. An empirical Bayes approach to statistics. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, volume 1, pages 157-163, 1956.
  31. Milton Rokeach. The role of values in public opinion research. Public Opinion Quarterly, 32(4):547-559, 1968.
  32. Magnus Roos, Jörg Rothe, and Björn Scheuermann. How to calibrate the scores of biased reviewers by quadratic programming. In AAAI Conference on Artificial Intelligence, 2011.
  33. Nihar B. Shah, Sivaraman Balakrishnan, Joseph Bradley, Abhay Parekh, Kannan Ramchandran, and Martin J. Wainwright. Estimation from pairwise comparisons: Sharp minimax bounds with topology dependence. Journal of Machine Learning Research, 17(1):2049-2095, 2016.
  34. Nihar B. Shah, Sivaraman Balakrishnan, Adityanand Guntuboyina, and Martin J. Wainwright. Stochasti- cally transitive models for pairwise comparisons: Statistical and computational issues. IEEE Transactions on Information Theory, 63(2):934-959, 2017.
  35. Nihar B. Shah, Joseph K Bradley, Abhay Parekh, Martin Wainwright, and Kannan Ramchandran. A case for ordinal peer-evaluation in MOOCs. In NIPS Workshop on Data Driven Education, 2013.
  36. Nihar B. Shah, Sivaraman Balakrishnan, and Martin J. Wainwright. A permutation-based model for crowd labeling: Optimal estimation and robustness. preprint arXiv:1606.09632, 2016. · arXiv
  37. Nihar B. Shah, Sivaraman Balakrishnan, and Martin J. Wainwright. Low permutation-rank matrices: Structural properties and noisy completion. In International Symposium on Information Theory, 2018.
  38. Nihar Shah. Learning From People. PhD thesis, EECS Department, University of California, Berkeley, 2017.
  39. Ivan Stelmakh, Nihar Shah, and Aarti Singh. PeerReview4All: Fair and accurate reviewer assignment in peer review. arXiv preprint arxiv:1806.06237, 2018. · arXiv
  40. Charles Stein. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, volume 1, pages 197-206, 1956.
  41. Nihar B. Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike Von Luxburg. Design and analysis of the NIPS 2016 review process. preprint arXiv:1708.09794, 2017. · arXiv
  42. Nihar B. Shah and Martin J. Wainwright. Simple, robust and optimal ranking from pairwise comparisons. Journal of Machine Learning Research, 2018.
  43. Kevin Tian, Weihao Kong, and Gregory Valiant. Learning populations of parameters. In Advances in Neural Information Processing Systems, 2017.
  44. Yichong Xu, Han Zhao, Xiaofei Shi, and Nihar Shah. On strategyproof conference review. arXiv preprint arxiv:1806.06266, 2018. · arXiv
  45. H. P. Young. Condorcet's theory of voting. American Political Science Review, 82(4):1231--1244, 1988.

Cited by

See all 4

Metadata

Authors
Wang, J., & Shah, N. B.
Venue
International Joint Conference on Autonomous Agents and Multiagent Systems
Publisher
IEEE Computer Society
Published
2019-05-08
Pages
864-872
ISSN
1558-2914
DOI
10.65109/gwnx2060
Cited by
4
References
1

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings. https://app.cochat.ai/papers/10.65109/gwnx2060
The paper
Wang, J., & Shah, N. B. (2019). Your 2 is My 1, Your 3 is My 9: Handling Arbitrary Miscalibrations in Ratings. International Joint Conference on Autonomous Agents and Multiagent Systems. https://doi.org/10.65109/gwnx2060

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.