Comparing Test Sets with Item Response Theory
Open accessAbstract
Clara Vania, Phu Mon Htut, William Huang, Dhara Mungra, Richard Yuanzhe Pang, Jason Phang, Haokun Liu, Kyunghyun Cho, Samuel R. Bowman. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Vania et al. employs IRT models in language model benchmarking to study saturation of commonly used benchmarks.
“Vania et al. (2021) employs IRT models in the context of language models benchmarking to study saturation (un-discriminability) of commonly used benchmarks”
From tinyBenchmarks: evaluating LLMs with fewer examples (2024) · cites this paper as Vania et al. (2021)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
Unsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
Self-citationA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · see citing works
Self-citationA Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · see citing works
Self-citationCommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1421 · Cited by 2,805 · see citing works
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · see citing works
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674 · see citing works
(2018) · International Conference on Learning Representations · Cited by 1,621 · see citing works
Self-citationDeBERTa: Decoding-enhanced BERT with Disentangled AttentionPreprint
He, P., Liu, X., Gao, J., & Chen, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03654 · Cited by 1,224 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., D., M., Châtel, G., Chaumond, J., Rault, T., Voss, C., Wang, F., et al. (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3385997 · Cited by 1,054 · see citing works
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works
Self-citationRACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044 · see citing works
Self-citationLevesque, H. J., Davis, E., & Morgenstern, L. (2011) · Cited by 879 · see citing works
Item Response Theory: Parameter Estimation Techniques.
Ackerman, T. A., & Baker, F. B. (1993) · Journal of the American Statistical Association · doi:10.2307/2290371 · Cited by 835 · see citing works
HellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830 · see citing works
Pyro: Deep Universal Probabilistic ProgrammingPreprint
Bingham, E., Chen, J. P., Jankowiak, M., Obermeyer, F. H., Pradhan, N., Karaletsos, T., Singh, R., Szerlip, P., Horsfall, P., & Goodman, N. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.09538 · Cited by 767 · see citing works
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Zellers, R., Bisk, Y., Schwartz, R., & Choi, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1009 · Cited by 712 · see citing works
NewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · cited 2× · see citing works
“6 – NewsQA (Trischler et al., 2017) 76,568 4,343 4,293 EM 57.”
MethodAdversarial NLI: A New Benchmark for Natural Language Understanding
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., & Kiela, D. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.441 · Cited by 656 · see citing works
Black Box Variational InferencePreprint
Ranganath, R., Gerrish, S., & Blei, D. M. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1401.0118 · Cited by 646 · see citing works
Social IQa: Commonsense Reasoning about Social Interactions
Sap, M., Rashkin, H., Chen, D., LeBras, R., & Choi, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1454 · Cited by 588 · see citing works
The pascal recognising textual entailment challenge
Dagan, I., & Glickman, O. (2005) · Cited by 579 · see citing works
WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Sakaguchi, K., Le Bras, R., Bhagavatula, C., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6399 · Cited by 579 · see citing works
The Seventh PASCAL Recognizing Textual Entailment Challenge.
Bentivogli, L., Clark, P. E., Dagan, I., & Giampiccolo, D. (2008) · Cited by 533 · see citing works
The CommitmentBank: Investigating projection in naturally occurring discourse
de Marneffe, M.-C., Simons, M., & Tonhauser, J. (2019) · Digital Access to Libraries · doi:10.18148/sub/2019.v23i2.601 · Cited by 471 · see citing works
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462 · see citing works
Probing Neural Network Comprehension of Natural Language Arguments
Niven, T., & Kao, H.-Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1459 · Cited by 395 · see citing works
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 391 · see citing works
Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning
Roemmele, M., Bejan, C. A., & Gordon, A. S. (2011) · Cited by 334 · see citing works
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
Pilehvar, M. T., & Camacho-Collados, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1128 · Cited by 296 · see citing works
Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning
Huang, L., Bras, R. J. L., Bhagavatula, C., & Choi, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1243 · Cited by 278 · see citing works
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?
Pruksachatkun, Y., Phang, J., Liu, H., Htut, P. M., Zhang, X., Pang, R. Y., Vania, C., Kann, K., & Bowman, S. R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.467 · Cited by 229 · see citing works
Self-citationMRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., & Chen, D. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5801 · Cited by 220 · see citing works
Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning
Dasigi, P., Liu, N. F., Marasović, A., Smith, N. A., & Gardner, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1606 · Cited by 153 · see citing works
Zhou, B., Khashabi, D., Ning, Q., & Roth, D. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1332 · Cited by 137 · see citing works
The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants
Habernal, I., Wachsmuth, H., Gurevych, I., & Stein, B. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1175 · Cited by 124 · see citing works
Exploring and Predicting Transferability across NLP Tasks
Vu, T., Wang, T., Munkhdalai, T., Sordoni, A., Trischler, A., Mattarella-Micke, A., Maji, S., & Iyyer, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.635 · Cited by 118 · see citing works
MuTual: A Dataset for Multi-Turn Dialogue Reasoning
Cui, L., Wu, Y., Liu, S., Zhang, Y., & Zhou, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.130 · Cited by 110 · see citing works
What Makes Reading Comprehension Questions Easier?
Sugawara, S., Inui, K., Sekine, S., & Aizawa, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1453 · Cited by 105 · see citing works
Item response theory in AI: Analysing machine learning classifiers at the instance level
Martínez‐Plumed, F., Prudêncio, R. B. C., Martínez-Usó, A., & Hernández‐Orallo, J. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.09.004 · Cited by 101 · see citing works
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
Wang, A., Hůla, J., Xia, P., Pappagari, R. R., McCoy, R. T., Patel, R., Kim, N., Tenney, I. F., Huang, Y., Yu, K., et al. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1439 · Cited by 101 · see citing works
Self-citationMCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge
Ostermann, S., Modi, A., Roth, M., Thater, S., & Pinkal, M. (2018) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2x36g84ob95p · Cited by 95 · see citing works
When Do You Need Billions of Words of Pretraining Data?
Zhang, Y., Warstadt, A., Li, X., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.90 · Cited by 85 · see citing works
Self-citationCrowdsourcing Question-Answer Meaning Representations
Michael, J., Stanovsky, G., He, L., Dagan, I., & Zettlemoyer, L. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2089 · Cited by 69 · see citing works
Getting Closer to AI Complete Question Answering: A Set of Prerequisite Real Tasks
Rogers, A., Kovaleva, O., Downey, M., & Rumshisky, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6398 · Cited by 68 · see citing works
jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models
Pruksachatkun, Y., Yeres, P., Liu, H., Phang, J., Htut, P. M., Wang, A., Tenney, I., & Bowman, S. R. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.15 · Cited by 67 · see citing works
Self-citationBuilding an Evaluation Scale using Item Response Theory
Lalor, J., Wu, H., & yu, H. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1062 · Cited by 60 · see citing works
Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark
Nangia, N., & Bowman, S. R. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1449 · Cited by 57 · see citing works
Self-citationEvaluation Examples are not Equally Informative: How should that change NLP Leaderboards?
Rodriguez, P., Barrow, J., Hoyle, A. M., Lalor, J. P., Jia, R., & Boyd-Graber, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.346 · Cited by 47 · see citing works
PIQA: Reasoning about Physical Commonsense in Natural Language
Bisk, Y., Zellers, R., Le bras, R., Gao, J., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6239 · Cited by 37 · see citing works
Lalor, J. P., Wu, H., & Yu, H. Q. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1434 · Cited by 31 · see citing works
Item Response Theory for Efficient Human Evaluation of Chatbots
Sedoc, J., & Ungar, L. (2020) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2020.eval4nlp-1.3 · Cited by 22 · see citing works
Lalor, J. P., Wu, H., Munkhdalai, T., & Yu, H. Q. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1500 · Cited by 22 · see citing works
A Framework for Evaluation of Machine Reading Comprehension Gold StandardsPreprint
Schlegel, V., Valentino, M., Freitas, A., Nenadić, G., & Batista-Navarro, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04642 · Cited by 21 · see citing works
Dynamic Data Selection for Curriculum Learning via Ability Estimation
Lalor, J. P., & Yu, H. Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.48 · Cited by 17 · see citing works
IRT-based Aggregation Model of Crowdsourced Pairwise Comparison for Evaluating Machine Translations
Otani, N., Nakazawa, T., Kawahara, D., & Kurohashi, S. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1049 · Cited by 14 · see citing works
Models of Translation Competitions
Hopkins, M., & May, J. (2013) · Cited by 13 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Han- nah Rashkin, Doug Downey, Wen tau Yih, and Yejin Choi. 2020. Abductive Commonsense Reasoning. In International Conference on Learning Represen- tations.
- Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recogniz- ing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1-9, Prague. Association for Computational Linguistics.
- In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Gar- nett, editors, Advances in Neural Information Pro- cessing Systems 32, pages 8024-8035.
Cited by
Sorted by citations
Dehghani, M., Tay, Y., Gritsenko, A. A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., & Vinyals, O. (2021) · arXiv · doi:10.48550/arxiv.2107.07002 · Cited by 48 · see citing works
“…2021, Bouthillier et al., 2021, Recht et al., 2019, Lin et al., 2021, Balduzzi et al., 2018, Lipton and Steinhardt, 2018, Bouthillier et al., 2019, Vania et al., 2020, Mishra and Arunkumar, 2021, Marie et al., 2021, Dodge et al., 2019] are of utmost importance to correct empirical evaluation of…”
BackgroundEvaluation Examples are not Equally Informative: How should that change NLP Leaderboards?
Rodriguez, P., Barrow, J., Hoyle, A. M., Lalor, J. P., Jia, R., & Boyd-Graber, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.346 · Cited by 47 · see citing works
Nangia, N., Sugawara, S., Trivedi, H. K., Warstadt, A., Vania, C., & Bowman, S. R. (2021) · doi:10.18653/v1/2021.acl-long.98 · Cited by 25 · see citing works
“Following Vania et al. (2021), we use 90 Transformer-based models fine-tuned on RACE, with varying ability levels, and use their predictions on our four datasets as responses.”
MethodSelf-citationModels in the Loop: Aiding Crowdworkers with Generative Annotation Assistants
Bartolo, M., Thrush, T., Riedel, S., Stenetorp, P., Jia, R., & Kiela, D. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.275 · Cited by 19 · see citing works
ILDAE: Instance-Level Difficulty Analysis of Evaluation Data
Varshney, N., Mishra, S. R., & Baral, C. R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.240 · Cited by 17 · see citing works
Predicting Difficulty and Discrimination of Natural Language Questions
Byrd, M., & Srivastava, S. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.15 · Cited by 10 · see citing works
Predicting Human Psychometric Properties Using Computational Language Models
Laverghetta, A., Nighojkar, A., Mirzakhalov, J., & Licato, J. (2022) · Springer proceedings in mathematics & statistics · doi:10.1007/978-3-031-04572-1_12 · Cited by 11 · see citing works
tinyBenchmarks: evaluating LLMs with fewer examples
Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M. (2024) · arXiv · doi:10.48550/arxiv.2402.14992 · Cited by 8 · see citing works
“For instance, Lalor et al. (2016) propose using IRT’s latent variables to measure language model abilities, Vania et al. (2021) employs IRT models in the context of language models benchmarking to study saturation (un-discriminability) of commonly used benchmarks, and Rodriguez et al. (2021) study…”
MethodAdversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Fair
Phang, J., Chen, A., Huang, W., & Bowman, S. R. (2022) · doi:10.18653/v1/2022.dadc-1.8 · Cited by 8 · see citing works
Self-citationGeneral scales unlock AI evaluation with explanatory and predictive power
Zhou, L., Pacchiardi, L., Martínez‐Plumed, F., Collins, K. M., Moros-Daval, Y., Zhang, S., Zhao, Q., Huang, Y., Sun, L., Prunty, J. E., et al. (2026) · Nature · doi:10.1038/s41586-026-10303-2 · Cited by 8 · see citing works
“• Measurements robust to changing populations (system process only, if reusing ADeLe: usually, benchmarks are replaced whenever a relevant part of the population of AI systems to which they are applicable achieves accuracy equally close to the maximum (termed ‘satura-tion’ 98 ).”
Investigating UD Treebanks via Dataset Difficulty Measures
Kulmizev, A., & Nivre, J. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.76 · Cited by 4 · see citing works
“If working with a wide range of parsing models, one could employ item response theory (Rodriguez et al., 2021; Vania et al., 2021), which is a framework that consolidates many predictions per instance in order to identify regions of datasets that may be perceived as difficult, easy, etc.”
BackgroundMethodAnalysing the Predictability of Language Model Performance
Schellaert, W., Martínez‐Plumed, F., & Hernández‐Orallo, J. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3706118 · Cited by 4 · see citing works
“For discussions and use in ML see [35–37] or [48] for natural language processing (NLP) in particular.”
On the data requirements of probing
Zhu, Z., Wang, J., Li, B., & Rudzicz, F. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.326 · Cited by 5 · see citing works
Quantifying Variance in Evaluation Benchmarks
Madaan, L., Singh, A. K., Schaeffer, R., Poulton, A., Koyejo, S., Stenetorp, P., Narang, S., & Hupkes, D. (2024) · arXiv · doi:10.48550/arxiv.2406.10229 · Cited by 2 · see citing works
“In the recent past, the method has become popular as a means of understanding model performance on a set of evaluation samples (Lalor et al., 2016; Vania et al., 2021; Rodriguez et al., 2021).”
fl-IRT-ing with Psychometrics to Improve NLP Bias Measurement
Bachmann, D., van der Wal, O., Chvojka, E., Zuidema, W. H., van Maanen, L., & Schulz, K. (2024) · Minds and Machines · doi:10.1007/s11023-024-09695-9 · Cited by 1 · see citing works
“Secondly, Vania et al. (2021) recently introduced (item characteristic curve-derived) Locally Estimated Headroom scores , a measure of how likely benchmark datasets are to (not) be trivially easy for the next generation of state-of-the-art language models.”
InfluentialClustering Examples in Multi-Dataset Benchmarks with Item Response Theory
Rodríguez, P., Htut, P. M., Lalor, J., & Sedoc, J. (2022) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2022.insights-1.14 · Cited by 1 · see citing works
“In NLP, IRT is used to identify helpful training examples (Lalor and Yu, 2020), detect errors in evaluation examples (Rodriguez et al., 2021), and estimate the future utility of examples in benchmarks (Vania et al., 2021).”
BackgroundMethodInfluentialSelf-citationFeinglass, J., & Yang, Y. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00434 · Cited by 1 · see citing works
“…Item Response Theory (IRT) is a test example selection and weighting mechanism gaining popularity in Natural Language Processing benchmarks [20, 35, 41] which seeks to provide greater rewards for more difficult text examples and has been shown to be more reliable and representative than standard…”
BackgroundOn Degrees of Freedom in Defining and Testing Natural Language Understanding
Sugawara, S., & Tsugita, S. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.861 · Cited by 1 · see citing works
LiveRAG: A Diverse Q&A Dataset with Varying Difficulty Level for RAG Evaluation
Carmel, D., Filice, S., Horowitz, G., Maarek, Y., Shtoff, A., Somekh, O., & Tavory, R. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808585
“We analyze the benchmark characteristics using Item Response Theory (IRT), a framework from psychometrics [18, 26, 33], which jointly estimates latent traits of questions and of participating systems (subjects) in the LiveRAG challenge.”
Diagnosing LLM Benchmark: A Psychometric Analysis of Difficulty and Discrimination
Qin, J., Zhang, X., Feng, D., Ding, B., & Zhai, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-31673-8_10
- Authors
- Vania, C., Htut, P. M., Huang, W., Mungra, D., Pang, R. Y., Phang, J., Liu, H., Cho, K., & Bowman, S. R.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- New York University (US)
- Venue
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Published
- 2021
- Pages
- 1141-1158
- OpenAlex
- W3171135604
- Cited by
- 18
- References
- 62
- Citations by year
- 2026: 3 · 2024: 3 · 2023: 2 · 2022: 7 · 2021: 3
- FWCI
- 1.73
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- arXiv (Cornell University) (submittedVersion)
- Faculty Digital Archive (New York University Florence) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1922658)
- Nvidia
- Samsung
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- item response theory, benchmark evaluation, natural language processing benchmarks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
