Building an Evaluation Scale using Item Response Theory
Open accessAbstract
Evaluation of NLP methods requires testing against a previously vetted gold-standard test set and reporting standard metrics (accuracy/precision/recall/F1). The current assumption is that all items in a given test set are equal with regards to difficulty and discriminating power. We propose Item Response Theory (IRT) from psychometrics as an alternative means for gold-standard test-set generation and NLP system evaluation. IRT is able to describe characteristics of individual items - their difficulty and discriminating power - and can account for these characteristics in its estimation of human intelligence or ability for an NLP task. In this paper, we demonstrate IRT by generating a gold-standard test set for Recognizing Textual Entailment. By collecting a large number of human responses and fitting our IRT model, we show that our IRT model compares NLP systems with the performance in a human population and is able to provide more insight into system performance than standard evaluation metrics. We show that a high accuracy score does not always imply a high IRT score, which depends on the item characteristics and the response pattern.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Lalor et al. propose using IRT's latent variables to measure language model abilities.
“For instance, Lalor et al. (2016) propose using IRT's latent variables to measure language model abilities”
From tinyBenchmarks: evaluating LLMs with fewer examples (2024) · cites this paper as Lalor et al. (2016)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 28 · Cited by 60
Research works printed but not linked: 13
- An Joohui Lee Seungwoo Lee Gary Geunbae Automatic Acquisition of Named Entity Tagged Corpus from World Wide Web 2003 Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 2, ACL ’03 Stroudsburg, PA, USA Association for Computational Linguistics 165 168
- Baker Frank B Kim Seock-Ho Item Response Theory: Parameter Estimation Techniques 2004 7 Second CRC Press
- Beltagy Islam Roller Stephen Cheng Pengxiang Erk Katrin Mooney Raymond J Representing Meaning with a Combination of Logical Form and Vectors arXiv: 1505.06816 [cs] 2015 arXiv: 1505.06816 · arXiv
- Bowman Samuel R Angeli Gabor Potts Christopher Manning Christopher D A large annotated corpus for learning natural language inference Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP) 2015 Association for Computational Linguistics
- Chen Wen-Hung Thissen David Local Dependence Indexes for Item Pairs Using Item Response Theory Journal of Educational and Behavioral Statistics 1997 9 22 3 265 289
- Dagan Ido Glickman Oren Magnini Bernardo The PASCAL Recognising Textual Entailment Challenge Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment 2006 Springer 177 190
- Guo Weiwei Li Hao Ji Heng Diab Mona T Linking Tweets to News: A Framework to Enrich Short Text Data in Social Media ACL (1) 2013 Citeseer 239 249
- Hovy Dirk Berg-Kirkpatrick Taylor Vaswani Ashish Hovy Eduard H Learning whom to trust with mace HLT-NAACL 2013 1120 1130
- Levy Omar Dagan Ido Goldberger Jacob Focused entailment graphs for open IE propositions Proc. CoNLL 2014
- Roller Roland Stevenson Mark Held-out versus Gold Standard: Comparison of Evaluation Strategies for Distantly Supervised Relation Extraction from Medline abstracts Sixth International Workshop on Health Text Mining and Information Analysis (LOUHI) 2015 97
- Silver David Huang Aja Maddison Chris J Guez Arthur Sifre Laurent van den Driessche George Schrittwieser Julian Antonoglou Ioannis Panneershelvam Veda Lanctot Marc Mastering the game of go with deep neural networks and tree search Nature 2016 529 7587 484 489 26819042
- Wiebe Janyce M Bruce Rebecca F O’Hara Thomas P Development and use of a gold-standard data set for subjectivity classifications Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics 1999 Association for Computational Linguistics 246 253
- Young Peter Lai Alice Hodosh Micah Hockenmaier Julia From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions Transactions of the Association for Computational Linguistics 2014 2 2 0 67 78
- Authors
- Lalor, J., Wu, H., & yu, H.
- Affiliations
- Boston College (US)
- Edith Nourse Rogers Memorial Veterans Hospital (US)
- Venue
- Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
- Published
- 2016
- Volume
- 2016
- Pages
- 648-657
- PMC
- PMC5167538
- OpenAlex
- W2963677523
- Cited by
- 60
- References
- 28
- Citations by year
- 2026: 2 · 2025: 7 · 2024: 6 · 2023: 6 · 2022: 3 · 2021: 15 · 2020: 13 · 2019: 3 · 2018: 3 · 2017: 2
- FWCI
- 1.44
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- U.S. Department of Veterans Affairs (I01 HX001457)
- National Institutes of Health
- National Heart, Lung, and Blood Institute (R01HL125089)
- U.S. National Library of Medicine (R01LM012817)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- item response theory, natural language inference, item difficulty, item discrimination, psychometrics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
