Anchor Points: Benchmarking Models with Much Fewer Examples
Open accessAbstract
Modern language models often exhibit powerful but brittle behavior, leading to the development of larger and more diverse benchmarks to reliably assess their behavior. Here, we suggest that model performance can be benchmarked and elucidated with much smaller evaluation sets. We first show that in six popular language classification benchmarks, model confidence in the correct class on many pairs of points is strongly correlated across models. We build upon this phenomenon to propose Anchor Point Selection, a technique to select small subsets of datasets that capture model behavior across the entire dataset. Anchor points reliably rank models: across 87 diverse language model-prompt pairs, evaluating models using 1-30 anchor points outperforms uniform sampling and other baselines at accurately ranking models. Moreover, just several anchor points can be used to estimate model per-class predictions on all other points in a dataset with low mean absolute error, sufficient for gauging where the model is likely to fail. Lastly, we present Anchor Point Maps for visualizing these insights and facilitating comparisons of the performance of different models on various regions within the dataset distribution.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Clustering examples based on LLMs that have already been evaluated performs well in some settings but can be unreliable when correctness patterns are spurious.
“Clustering examples based on LLMs that have already been evaluated.The key idea is to find examples where (in)correct prediction of an LLM implies that it will also be (in)correct on a subset of other examples.This method performs well in some settings but can be unreliable when such correctness patterns are spurious, e.g., when predicting the accuracy of an LLM specialized to a domain.”
From tinyBenchmarks: evaluating LLMs with fewer examples (2024) · cites this paper as Vivek et al. (2023)
The Introduction (page unavailable) describes selecting examples using correlations in previously evaluated models’ predictions, and the Multitask Language Understanding section reports good results on several datasets but poor results on Global Facts, where learned correlations may be spurious. The paper does not identify predicting the accuracy of a domain-specialized LLM as an example of this failure. (LLM, conf=0.86)
Experimental — accuracy not yet measuredVivek et al. proposed clustering evaluation examples based on models' confidence in the correct class for faster evaluation on classification tasks.
“Vivek et al. (2023) proposed clustering evaluation examples based on models' confidence in the correct class for faster evaluation on classification tasks.”
From tinyBenchmarks: evaluating LLMs with fewer examples (2024) · cites this paper as Vivek et al. (2023)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · cited 1× · see citing works
“For baselines, we use uniform random sampling as well as K-Medoids sampling over the embedding spaces of a generic sentence encoder– Sentence-BERT (Reimers and Gurevych., 2019) – and an encoder fine-tuned on the dataset– the CLS token of bert-base-uncased (Devlin et al., 2018).”
MethodSentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 1× · see citing works
“For baselines, we use uniform random sampling as well as K-Medoids sampling over the embedding spaces of a generic sentence encoder– Sentence-BERT (Reimers and Gurevych., 2019) – and an encoder fine-tuned on the dataset– the CLS token of bert-base-uncased (Devlin et al., 2018).”
MethodFinding Groups in Data: An Introduction to Cluster Analysis
Kaufman, L., & Rousseeuw, P. J. (1990) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316801 · Cited by 7,344 · cited 2× · see citing works
“We efficiently solve this objective using the Partitioning Around Medoids (PAM) al-gorithm (Kaufman and Rousseeuw, 2009).”
MethodUMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
McInnes, L., Healy, J., & Melville, J. (2018) · arXiv · doi:10.48550/arxiv.1802.03426 · Cited by 7,101 · cited 2× · see citing works
“Finally, we cast the points to a high-dimensional continuous space using Multi-Dimensional Scaling on Z and project the space to two components with U-MAP (McInnes et al., 2018).”
MethodRecursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1× · see citing works
“Yet these same systems prove to be brittle, spurring the development of larger, more diverse, and more shrewd benchmarks to reliably assess their behavior (Rajpurkar et al., 2016; Hendrycks et al., 2021; Kiela et al., 2021; Wu et al., 2023).”
BackgroundGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 1× · see citing works
“For six datasets from the GLUE benchmark (Wang et al., 2019), we finetune each BERT-family model on the training split D train and obtain its predicted probability of the correct class on each instance in the validation split D eval .”
MethodSimple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666 · see citing works
A theory of learning from different domains
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., & Vaughan, J. W. (2010) · Machine Learning · doi:10.1007/s10994-009-5152-4 · Cited by 3,515 · cited 2× · see citing works
“This unexplained by classical theory which provides only weak bounds relating the two metrics (Ben-David et al., 2018).”
BackgroundA Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · see citing works
DeBERTa: Decoding-enhanced BERT with Disentangled AttentionPreprint
He, P., Liu, X., Gao, J., & Chen, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03654 · Cited by 1,224 · see citing works
Information-based objective functions for active data selection
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.4.590 · Cited by 1,221 · cited 1× · see citing works
“We observe that all datasets overlap, but points tend to cluster with other points from the same dataset or a related dataset. model predictions (MacKay, 1992; Kirsch et al., 2021).”
BackgroundA Survey of Deep Active Learning
Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Gupta, B. B., Chen, X., & Wang, X. (2022) · ACM Computing Surveys · doi:10.1145/3472291 · Cited by 1,169 · see citing works
Neural Network Acceptability Judgments
Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00290 · Cited by 984 · see citing works
BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active LearningPreprint
Kirsch, A., van Amersfoort, J. R., & Gal, Y. (2019) · Neural Information Processing Systems · doi:10.48550/arxiv.1906.08158 · Cited by 826 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Self-citationBayesian Active Learning for Classification and Preference Learning
Houlsby, N., Huszár, F., Ghahramani, Z., & Lengyel, M. (2011) · arXiv · doi:10.48550/arxiv.1112.5745 · Cited by 507 · see citing works
Knapp, W. (2019) · CIRP Encyclopedia of Production Engineering · doi:10.1007/978-3-662-53120-4_6575 · Cited by 423 · cited 5×
“Models Miller et al. (2021) present the Accuracy on the Line phenomenon: out-of-distribution (OOD) performance is strongly positively correlated with in-distribution (ID) performance for a wide range of models and distribution shifts.”
BackgroundMethodInfluentialData Selection for Language Models via Importance ResamplingPreprint
Xie, S. M., Santurkar, S., Ma, T.-Y., et al. (2023) · Neural Information Processing Systems · doi:10.48550/arxiv.2302.03169 · Cited by 397 · see citing works
GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model TrainingPreprint
Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., et al. (2021) · International Conference on Machine Learning · doi:10.48550/arxiv.2103.00123 · Cited by 386 · cited 1× · see citing works
“…techniques including clustering (Kaufman and Rousseeuw, 2009; Reza Zan-jirani Farahani, 2009), gradient-matching (Mirza-soleiman et al., 2020; Killamsetty et al., 2021a), bi-level optimization (Borsos et al., 2020; Killam-setty et al., 2021b), and submodularity-based meth-ods (Kai Wei, 2015).”
Understanding Dataset Difficulty with V-Usable InformationPreprint
Ethayarajh, K., Choi, Y., & Swayamdipta, S. (2021) · International Conference on Machine Learning · doi:10.48550/arxiv.2110.08420 · Cited by 376 · cited 2× · see citing works
“Ethayarajh et al. (2022) further show that instance-level predictions can be used to quantify how much information a given model can extract from a dataset, providing a formal metric of dataset difficulty that exposes model behavior at the dataset and data point level.”
BackgroundSelf-citationDataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics
Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H., Smith, N. A., & Choi, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.746 · Cited by 320 · cited 1× · see citing works
“Swayamdipta et al. (2020) present Data Maps, a powerful technique that leverages instance-level predictions to reveal underlying structure in the interplay of models and data points.”
MethodMeasuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · cited 4× · see citing works
“Yet these same systems prove to be brittle, spurring the development of larger, more diverse, and more shrewd benchmarks to reliably assess their behavior (Rajpurkar et al., 2016; Hendrycks et al., 2021; Kiela et al., 2021; Wu et al., 2023).”
BackgroundMethodInfluentialAutomatic Chain of Thought Prompting in Large Language Models
Zhang, Z., Zhang, A., Li, M., & Smola, A. (2022) · arXiv · doi:10.48550/arxiv.2210.03493 · Cited by 274 · see citing works
Dynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254 · see citing works
Self-citationDeepCore: A Comprehensive Library for Coreset Selection in Deep Learning
Guo, C., Zhao, B., & Bai, Y. (2022) · International Conference on Database and Expert Systems Applications · doi:10.48550/arxiv.2204.08499 · Cited by 246 · cited 1×
“Coresets Anchor Points can be interpreted as a coreset, broadly defined as a small "summary" of a large set of data (Feldman, 2020; Guo et al., 2022).”
BackgroundTrue Few-Shot Learning with Language Models
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv · doi:10.48550/arxiv.2105.11447 · Cited by 226 · see citing works
Self-citationPromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
Bach, S., Sanh, V., Yong, Z. X., Webson, A., Raffel, C., Nayak, N. V., Sharma, A., Kim, T., Bari, M. S., Fevry, T., et al. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.9 · Cited by 178 · cited 1× · see citing works
“All prompts are randomly selected from Prompt Source (Bach et al., 2022) QQP 1.”
MethodGLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning
Killamsetty, K., Sivasubramanian, D., Ramakrishnan, G., & Iyer, R. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i9.16988 · Cited by 135 · cited 1× · see citing works
“…techniques including clustering (Kaufman and Rousseeuw, 2009; Reza Zan-jirani Farahani, 2009), gradient-matching (Mirza-soleiman et al., 2020; Killamsetty et al., 2021a), bi-level optimization (Borsos et al., 2020; Killam-setty et al., 2021b), and submodularity-based meth-ods (Kai Wei, 2015).”
Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift
Baek, C., Jiang, Y., Raghunathan, A., et al. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2206.13089 · Cited by 124 · cited 5×
“Models Miller et al. (2021) present the Accuracy on the Line phenomenon: out-of-distribution (OOD) performance is strongly positively correlated with in-distribution (ID) performance for a wide range of models and distribution shifts.”
BackgroundInfluentialActive Testing: Sample-Efficient Model EvaluationPreprint
Kossen, J., Farquhar, S., Gal, Y., & Rainforth, T. (2021) · International Conference on Machine Learning · doi:10.48550/arxiv.2103.05331 · Cited by 123 · cited 1× · see citing works
“Many works (Kossen et al., 2021a,b; Deng and Zheng, 2021; Corneanu et al., 2020) minimize evaluation annotation costs by actively selecting points to annotate for evaluating a given model.”
BackgroundSelf-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2022) · arXiv · doi:10.48550/arxiv.2212.10560 · Cited by 118 · see citing works
What Will it Take to Fix Benchmarking in Natural Language Understanding?
Bowman, S. R., & Dahl, G. E. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.385 · Cited by 117 · cited 1× · see citing works
“Other works (Rodriguez et al., 2021; Ethayarajh et al., 2022; Bowman and Dahl, 2020) point out that it is often a minority of points that differentiate the performance of various models.”
BackgroundOn Statistical Bias In Active Learning: How and When To Fix ItPreprint
Farquhar, S., Gal, Y., & Rainforth, T. (2021) · International Conference on Learning Representations · doi:10.48550/arxiv.2101.11665 · Cited by 106 · see citing works
Coresets via Bilevel Optimization for Continual Learning and StreamingPreprint
Borsos, Z., Mutný, M., & Krause, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03875 · Cited by 102 · cited 1× · see citing works
“…techniques including clustering (Kaufman and Rousseeuw, 2009; Reza Zan-jirani Farahani, 2009), gradient-matching (Mirza-soleiman et al., 2020; Killamsetty et al., 2021a), bi-level optimization (Borsos et al., 2020; Killam-setty et al., 2021b), and submodularity-based meth-ods (Kai Wei, 2015).”
Are Labels Always Necessary for Classifier Accuracy Evaluation?
Deng, W., & Zheng, L. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01482 · Cited by 91 · cited 1× · see citing works
“Many works (Kossen et al., 2021a,b; Deng and Zheng, 2021; Corneanu et al., 2020) minimize evaluation annotation costs by actively selecting points to annotate for evaluating a given model.”
BackgroundLogME: Practical Assessment of Pre-trained Models for Transfer LearningPreprint
You, K., Liu, Y., Wang, J., & Long, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.11005 · Cited by 87 · see citing works
With Little Power Comes Great Responsibility
Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.745 · Cited by 84 · see citing works
Introduction to Core-sets: an Updated Survey
Feldman, D. (2020) · arXiv.org · doi:10.48550/arxiv.2011.09384 · Cited by 83 · cited 1×
“Coresets Anchor Points can be interpreted as a coreset, broadly defined as a small "summary" of a large set of data (Feldman, 2020; Guo et al., 2022).”
BackgroundComputing the Testing Error Without a Testing Set
Corneanu, C., Madadi, M., Escalera, S., et al. (2020) · Computer Vision and Pattern Recognition · doi:10.1109/cvpr42600.2020.00275 · Cited by 81 · cited 1× · see citing works
“Many works (Kossen et al., 2021a,b; Deng and Zheng, 2021; Corneanu et al., 2020) minimize evaluation annotation costs by actively selecting points to annotate for evaluating a given model.”
BackgroundImitation Attacks and Defenses for Black-box Machine Translation Systems
Wallace, E., Stern, M., & Song, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.446 · Cited by 75 · see citing works
Miller, J., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., & Schmidt, L. (2021) · arXiv · doi:10.48550/arxiv.2107.04649 · Cited by 74 · cited 6× · see citing works
“This mirrors phenomena like Accuracy on the Line (Miller et al., 2021) and Agreement on the Line (Baek et al.”
BackgroundMethodInfluentialSimfluence: Modeling the Influence of Individual Training Examples by Simulating Training RunsPreprint
Guu, K., Webson, A., Pavlick, E., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2303.08114 · Cited by 57 · see citing works
Exploring the Limits of Large Scale Pre-trainingPreprint
Abnar, S., Dehghani, M., Neyshabur, B., & Sedghi, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02095 · Cited by 56 · see citing works
The False Promise of Imitating Proprietary LLMs
Gudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P., Levine, S., & Song, D. (2023) · arXiv · doi:10.48550/arxiv.2305.15717 · Cited by 48 · see citing works
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?
Rodriguez, P., Barrow, J., Hoyle, A. M., Lalor, J. P., Jia, R., & Boyd-Graber, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.346 · Cited by 47 · cited 2× · see citing works
“Other works (Rodriguez et al., 2021; Ethayarajh et al., 2022; Bowman and Dahl, 2020) point out that it is often a minority of points that differentiate the performance of various models.”
BackgroundZhang, M., & Plank, B. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.findings-emnlp.36 · Cited by 44 · see citing works
Language Models in the Loop: Incorporating Prompting into Weak Supervision
Smith, R., Fries, J., Hancock, B. J., & Bach, S. H. (2024) · ACM / IMS Journal of Data Science · doi:10.1145/3617130 · Cited by 39 · see citing works
Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation
Kossen, J., Farquhar, S., Gal, Y., et al. (2022) · Neural Information Processing Systems · doi:10.52202/068431-1783 · Cited by 37 · cited 1× · see citing works
“Many works (Kossen et al., 2021a,b; Deng and Zheng, 2021; Corneanu et al., 2020) minimize evaluation annotation costs by actively selecting points to annotate for evaluating a given model.”
Background(2020) · Future Theory · doi:10.32388/fae8q3 · Cited by 35
Kirsch, A., Rainforth, T., & Gal, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.11719 · Cited by 29 · cited 1× · see citing works
“We observe that all datasets overlap, but points tend to cluster with other points from the same dataset or a related dataset. model predictions (MacKay, 1992; Kirsch et al., 2021).”
BackgroundWang, C.-Q., Sun, S.-Y., & Grosse, R. (2020) · International Conference on Artificial Intelligence and Statistics · doi:10.48550/arxiv.2011.03178 · Cited by 29
Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level
Zhong, R., Ghosh, D., Klein, D., & Steinhardt, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.334 · Cited by 23 · cited 1× · see citing works
“Instance-Level Model Performance Despite being noisy (Zhong et al., 2021), instance-level model predictions are a rich source of information about model behavior.”
BackgroundActive Learning at the ImageNet ScalePreprint
Emam, Z., Chu, H. M., Chiang, P.-Y., Czaja, W., Leapman, R. D., Goldblum, M., & Goldstein, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12880 · Cited by 10 · see citing works
ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretation
Wu, Z., Manning, C. D., & Potts, C. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00623 · Cited by 9 · cited 1× · see citing works
“Yet these same systems prove to be brittle, spurring the development of larger, more diverse, and more shrewd benchmarks to reliably assess their behavior (Rajpurkar et al., 2016; Hendrycks et al., 2021; Kiela et al., 2021; Wu et al., 2023).”
BackgroundEvidence > Intuition: Transferability Estimation for Encoder Selection
Bassignana, E., Müller-Eberstein, M., Zhang, M., et al. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2210.11255 · Cited by 9
Flagged first, then by citations
28 of the 84 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Shai Ben-David and Shai Shalev-Shwartz. 2014. Un- derstanding Machine Learning: From Theory to Al- gorithms. Cambridge University Press.
- William B Dolan and Chris Brockett. 2004. Automati- cally constructing a corpus of sentential paraphrases. International Workshop on Paraphrasing.
- Leonard Kaufman and Peter J Rousseeuw. 2009. Find- ing groups in data: an introduction to cluster analy- sis. John Wiley & Sons.
- Masoud Hekmatfar Reza Zanjirani Farahani. 2009. Fa- cility location: concepts, models, algorithms and case studies. Springer-Verlag Berlin Heidelberg.
- Someone just said to me "{{sentence}}". Do you think they are {{"sad"}} or {{"happy"}}?' 3. I'm reading a review that says "{{sentence}}". Do you think the review is {{"positive"}} or {{"negative"}}? RTE
Cited by
Sorted by citations
How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench
Ye, Q., Fu, H. Y., Ren, X., & Jia, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.503 · Cited by 11 · see citing works
tinyBenchmarks: evaluating LLMs with fewer examples
Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M. (2024) · arXiv · doi:10.48550/arxiv.2402.14992 · Cited by 8 · see citing works
“This strategy is inspired by the Anchor Points method (Vivek et al., 2023) which clusters models’ confidence in the correct class for faster evaluation on classification tasks.”
MethodFlashEval: Towards Fast and Accurate Evaluation of Text-to-Image Diffusion Generative Models
Zhao, L. H., Zhao, T., Lin, Z., Ning, X., Dai, G., Yang, H., & Wang, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01526 · Cited by 5 · see citing works
Quantifying Variance in Evaluation Benchmarks
Madaan, L., Singh, A. K., Schaeffer, R., Poulton, A., Koyejo, S., Stenetorp, P., Narang, S., & Hupkes, D. (2024) · arXiv · doi:10.48550/arxiv.2406.10229 · Cited by 2 · see citing works
“With the aims of improving efficiency in model development cycles, recent work proposes reducing the size of evaluation benchmarks by picking representative samples (Vivek et al., 2023; Polo et al., 2024).”
Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data
Arviv, O., Greenewald, K., Perlitz, Y., Mulian, H., Shmueli-Scheuer, M., & Choshen, L. (2026) · Findings of the Association for Computational Linguistics: ACL 2026 · doi:10.18653/v1/2026.findings-acl.43
“Other approaches have proposed intelligent sampling methods to maintain reliable results (Polo et al., 2024a; Zhang et al., 2024; Vivek et al., 2024).”
Efficient Benchmarking via Bias-Bounded Subset Selection
Zhuang, Y., Yu, J., Liu, Q., Sun, Y., Li, J., Huang, Z., & Chen, E. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3598031
“One category clusters items in the benchmark and selects anchor or representative items [10], [28]; the other is adaptive testing, which dynamically selects informative items based on the model’s performance during the evaluation process [12], [13], [14].”
Adaptive LLM Selection for Question Answering over Historical Digital Libraries
Hamdi, A., Moreno, J. G., Armand, C., et al. (2026) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1145/3805696.3846025
- Authors
- Vivek, R., Ethayarajh, K., Yang, D., & Kiela, D.
- Affiliations
- Stanford University (US)
- ORCID
- Yang, Diyi
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 1576-1601
- arXiv
- 2309.08638
- OpenAlex
- W4411630215
- Semantic Scholar
- d4085ae0f004624a3141734d3a88a9ebbc803a55
- Cited by
- 4
- References
- 84
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 1 · 2023: 1
- FWCI
- 0.87
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference of the European Chapter of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- anchor point selection, language model benchmarking, model ranking, model confidence, model brittleness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
