The DOI matches its DataCite record.
Evaluating the Evaluators: Are Current Few-Shot Learning Benchmarks Fit for Purpose?
Open accessAbstract
Numerous benchmarks for Few-Shot Learning have been proposed in the last decade. However all of these benchmarks focus on performance averaged over many tasks, and the question of how to reliably evaluate and tune models trained for individual tasks in this regime has not been addressed. This paper presents the first investigation into task-level evaluation -- a fundamental step when deploying a model. We measure the accuracy of performance estimators in the few-shot setting, consider strategies for model selection, and examine the reasons for the failure of evaluators usually thought of as being robust. We conclude that cross-validation with a low number of folds is the best choice for directly estimating the performance of a model, whereas using bootstrapping or cross validation with a large number of folds is better for model selection purposes. Overall, we find that existing benchmarks for few-shot learning are not designed in such a way that one can get a reliable picture of how effectively methods can be used on individual tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Human voting has been shown to mitigate some biases associated with LLM-as-a-judge approaches.
“Human voting has been shown to mitigate some of the biases associated with using LLM-as-a-judge approaches”
From The Leaderboard Illusion (2025) · cites this paper as Shimabucoro et al., 2023
The Introduction and §§3.2–4.2 discuss biases in few-shot-learning performance estimators, not biases in LLM-as-a-judge approaches. Searches for human voting and human annotators judging language models found no passage addressing whether voting mitigates those biases. The indexed passages provide no page numbers. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 22
- Agarwal, M., Yurochkin, M., and Sun, Y. On sensitivity of meta-learning to support data. In Advances in Neural Information Processing Systems, 2021.
- Aghbalou, A., Portier, F., and Sabourin, A. Consistent cross validation with stable learners. arXiv preprint arXiv:2202.10211, 2022. · arXiv
- Basu, S., Stanley, M., Bronskill, J. F., Feizi, S., and Mas- siceti, D. Hard-Meta-Dataset++: Towards Understand- ing Few-Shot Performance on Difficult Tasks. In The Eleventh International Conference on Learning Repre- sentations, 2023.
- Bertinetto, L., Henriques, J. F., Torr, P. H., and Vedaldi, A. Meta-learning with differentiable closed-form solvers. In ICLR, 2019.
- Bousquet, O. and Elisseeff, A. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.
- Chen, W., Liu, Y., Kira, Z., Wang, Y. F., and Huang, J. A closer look at few-shot classification. ICLR, 2019.
- Dhillon, G. S., Chaudhari, P., Ravichandran, A., and Soatto, S. A baseline for few-shot image classification. In Inter- national Conference on Learning Representations, 2020.
- Efron, B. Bootstrap methods: another look at the jackknife. Springer, 1979.
- Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks. In ICML, 2017.
- Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, A. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9):5149-5169, 2021.
- Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B. Human-level concept learning through probabilistic pro- gram induction. Science, 350(6266):1332-1338, 2015.
- LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature, 521(7553):436-444, 2015.
- Ravi, S. and Larochelle, H. Optimization as a model for few- shot learning. In International conference on learning representations, 2017.
- Ren, M., Triantafillou, E., Ravi, S., Snell, J., Swersky, K., Tenenbaum, J. B., Larochelle, H., and Zemel, R. S. Meta- learning for semi-supervised few-shot classification. In International Conference on Learning Representations.
- Snell, J., Swersky, K., and Zemel, R. S. Prototypical net- works for few shot learning. In NIPS, 2017.
- Stanley, M., Bronskill, J. F., Maziarz, K., Misztela, H., Lanini, J., Segler, M., Schneider, N., and Brockschmidt, M. Fs-mol: A few-shot learning dataset of molecules. In Thirty-fifth Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 2), 2021.
- Stone, M. Cross-validatory choice and assessment of statis- tical predictions. Journal of the royal statistical society: Series B (Methodological), 36(2):111-133, 1974.
- Triantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Evci, U., Xu, K., Goroshin, R., Gelada, C., Swersky, K., Man- zagol, P.-A., et al. Meta-dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2020.
- Ullah, I., Carrión-Ojeda, D., Escalera, S., Guyon, I., Huis- man, M., Mohr, F., van Rijn, J. N., Sun, H., Vanschoren, J., and Vu, P. A. Meta-album: Multi-domain meta-dataset for few-shot image classification. Advances in Neural Information Processing Systems, 35:3232-3247, 2022.
- Varoquaux, G. and Cheplygina, V. Machine learning for medical imaging: methodological failures and recom- mendations for the future. NPJ digital medicine, 5(1):48, 2022.
- Wang, Y., Yao, Q., Kwok, J. T., and Ni, L. M. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
- Xie, A., Singh, A., Levine, S., and Finn, C. Few-shot goal inference for visuomotor learning and planning. In Conference on Robot Learning, pp. 40-52. PMLR, 2018.
Cited by
Sorted by citations
Singh, S., Nan, Y., Wang, A., D'Souza, D., Kapoor, S., Üstün, A., Koyejo, S., Deng, Y., Longpre, S., Smith, N. A., et al. (2025) · arXiv · doi:10.48550/arxiv.2504.20879 · Cited by 108
Metadata
- Authors
- Shimabucoro, L., Hospedales, T., & Gouk, H.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2307.02732
- OpenAlex
- W4383604682
- Cited by
- 4
- References
- 0
- Citations by year
- 2026: 2 · 2025: 2
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- few-shot learning, benchmark evaluation, cross-validation, bootstrapping, model selection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
