CoChat
Sign Up
Cited by 11

The DOI matches its Crossref record.

Elo Uncovered: Robustness and Best Practices in Language Model Evaluation

Boubdir, Meriem, Kim, Edward, Ermis, Beyza, Hooker, Sara, Fadaee, Marzieh·2024·Advances in Neural Information Processing Systems 37
DOI 10.52202/079017-3367
Open this paper in CoChat — free account

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.52202/079017-3367

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In principle, the BT model should still handle deprecation reliably because of the transitivity property.

    “However, in principle, the BT model should still handle this reliably because of the transitivity property (Boubdir et al., 2024).”

    From The Leaderboard Illusion (2025) · cites this paper as Boubdir et al., 2024

    Section 3.1 describes transitivity as a desirable property that can allow model comparisons without direct head-to-head evaluations. But the paper discusses Elo rather than the BT model, does not address deprecation, and reports circumstances in which Elo rankings violate transitivity. It therefore does not establish that BT would handle deprecation reliably. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 31

  1. Max Bartolo, A Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. Beat the ai: Investigating adversarial human annotation for reading comprehension. Transactions of the As- sociation for Computational Linguistics, 8:662-678, 2020. URL https://api.semanticschola r.org/CorpusID:211010520.
  2. Quentin Bertrand, Wojciech Marian Czarnecki, and Gauthier Gidel. On the limitations of the elo, real-world games are transitive, not additive. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 2905-2921. PMLR, 25-27 Apr 2023. URL https://proceedings.mlr.press/v206/bertrand23a.html.
  3. John J. Binder and Murray Findlay. The effects of the bosman ruling on national and club teams in europe. Journal of Sports Economics, 13:107-129, 2009. URL https://api.semanticschola r.org/CorpusID:153873146.
  4. Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Nazanin Mohammadi Sepahvand, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Tal Arbel, Chris Pal, Gael Varoquaux, and Pascal Vincent. Accounting for variance in machine learning benchmarks. In A. Smola, A. Dimakis, and I. Stoica (eds.), Proceedings of Machine Learning and Systems, volume 3, pp. 747-769, 2021. URL https:// proceedings.mlsys.org/paper/2021/file/cfecdb276f634854f3ef915e2e980c31-Paper.pdf.
  5. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction- finetuned language models, 2022. URL https://arxiv.org/abs/2210.11416. · arXiv
  6. Arman Dehpanah, Muheeb Faizan Ghori, Jonathan F. Gemmell, and Bamshad Mobasher. Evaluat- ing team skill aggregation in online competitive games. 2021 IEEE Conference on Games (CoG), pp. 01-08, 2021. URL https://api.semanticscholar.org/CorpusID:235593176.
  7. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023. · arXiv
  8. Chris Drummond. Replicability is not reproducibility: Nor is it good science. In Evaluation Meth- ods for Machine Learning Workshop at the 26th International Conference on Machine Learning, ICML, 2009.
  9. Aram Ebtekar and Paul Liu. Elo-mmr: A rating system for massive multiplayer competitions. In Proceedings of the Web Conference 2021, WWW '21, pp. 1772-1784, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383127. doi: 10.1145/3442381.3450091. URL https://doi.org/10.1145/3442381.3450091. · DOI
  10. Mark E Glickman. A comprehensive guide to chess ratings. American Chess Journal, pp. 59-102, 1995.
  11. Mark E Glickman. Parameter estimation in large dynamic paired comparison experiments. Applied Statistics, pp. 377-394, 1999.
  12. Mark E Glickman. Example of the glicko-2 system. Boston University, pp. 1-6, 2012.
  13. Steven N. Goodman, Daniele Fanelli, and John P. A. Ioannidis. What does research reproducibility mean? Science Translational Medicine, 8(341):341ps12-341ps12, 2016. ISSN 1946-6234. doi: 10.1126/scitranslmed.aaf5027. · DOI · Paper page
  14. Odd Erik Gundersen and Sigbjørn Kjensmo. State of the art: Reproducibility in artificial intelli- gence. In AAAI, 2018.
  15. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. CoRR, abs/1709.06560, 2017. · arXiv
  16. Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill™: A bayesian skill rating system. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006. URL https://proceedings.neurips.cc/paper_files/paper/2 006/file/f44ee263952e65b3610b8ba51229d1f9-Paper.pdf.
  17. Lars Magnus Hvattum and Halvard Arntzen. Using elo ratings for match result prediction in association football. International Journal of Forecasting, 26(3):460-470, 2010. URL https: //EconPapers.repec.org/RePEc:eee:intfor:v:26:y::i:3:p:460-470.
  18. Rudolf Kadlec, Ondrej Bajgar, and Jan Kleindienst. Knowledge base completion: Baselines strike back. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 69-74, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/ W17-2609. · DOI · Paper page
  19. Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi. Soda: Million-scale dialogue distillation with social commonsense contextualization. ArXiv, abs/2212.10465, 2022. · arXiv
  20. Christoph Leitner, Achim Zeileis, and Kurt Hornik. Forecasting sports tournaments by ratings of (prob)abilities: A comparison for the euro 2008. International Journal of Forecasting, 26(3): 471-481, 2010. ISSN 0169-2070. doi: https://doi.org/10.1016/j.ijforecast.2009.10.001. URL https://www.sciencedirect.com/science/article/pii/S0169207009001459. Sports Forecasting. · DOI
  21. Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1823- 1840, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/20 · DOI
  22. Pranava Madhyastha and Rishabh Jain. On model stability as a function of random seed. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 929-939, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/K19-1087. URL https://aclanthology.org/K19-1087. · DOI
  23. Gábor Melis, Chris Dyer, and P. Blunsom. On the state of the art of evaluation in neural language models. ArXiv, abs/1707.05589, 2018. · arXiv
  24. Prabhat Nagarajan, Garrett Warnell, and Peter Stone. The impact of nondeterminism on repro- ducibility in deep reinforcement learning. In 2nd Reproducibility in Machine Learning Workshop at ICML, July 2018.
  25. Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensitivity and generalization in neural networks: an empirical study. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=HJC2SzZCW.
  26. Robert R. Snapp and Gil I. Shamir. Synthesizing irreproducibility in deep networks. CoRR, abs/2102.10696, 2021. · arXiv
  27. Anders Søgaard, Sebastian Ebert, Jasmijn Bastings, and Katja Filippova. We need to talk about random splits. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1823-1832, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.eacl-main.156. URL https://aclanthology .org/2021.eacl-main.156. · DOI
  28. Cecilia Summers and Michael J. Dinneen. Nondeterminism and instability in neural network opti- mization. In International Conference on Machine Learning, 2021. URL https://api.semant icscholar.org/CorpusID:232146616.
  29. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4149-4158, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1421. URL https://aclanthology.org/N19-1421. · DOI
  30. Ben P. Wise. Elo ratings for large tournaments of software agents in asymmetric games. ArXiv, abs/2105.00839, 2021. URL https://api.semanticscholar.org/CorpusID:233481682. · arXiv
  31. Donglin Zhuang, Xingyao Zhang, Shuaiwen Leon Song, and Sara Hooker. Randomness in neural network training: Characterizing the impact of tooling. ArXiv, abs/2106.11872, 2021. URL https://api.semanticscholar.org/CorpusID:235593009. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Boubdir, M., Kim, E., Ermis, B., Hooker, S., & Fadaee, M.
Venue
Advances in Neural Information Processing Systems 37
Publisher
Neural Information Processing Systems Foundation, Inc. (NeurIPS)
Published
2024
Pages
106135-106161
DOI
10.52202/079017-3367
Cited by
11
References
0

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review