CoChat
Sign Up
PreprintCited by 2Machine Learning and Data ClassificationSurvey Sampling and Estimation TechniquesImbalanced Data Classification Techniques

A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimation

Open access
Fogliato, Riccardo, Patil, Pratik, Monfort, Mathew, Perona, Pietro·2024·arXiv
DOI 10.48550/arxiv.2406.07320arXiv 2406.07320
Open this paper in CoChat — free accountOpen PDF

Abstract

Model performance evaluation is a critical and expensive task in machine learning and computer vision. Without clear guidelines, practitioners often estimate model accuracy using a one-time completely random selection of the data. However, by employing tailored sampling and estimation strategies, one can obtain more precise estimates and reduce annotation costs. In this paper, we propose a statistical framework for model evaluation that includes stratification, sampling, and estimation components. We examine the statistical properties of each component and evaluate their efficiency (precision). One key result of our work is that stratification via k-means clustering based on accurate predictions of model performance yields efficient estimators. Our experiments on computer vision datasets show that this method consistently provides more precise accuracy estimates than the traditional simple random sampling, even with substantial efficiency gains of 10x. We also find that model-assisted estimators, which leverage predictions of model accuracy on the unlabeled portion of the dataset, are generally more efficient than the traditional estimates based solely on the labeled data.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2406.07320

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Practitioners often reduce costs by using fewer samples, but these heuristic approaches lack statistical guarantees.

    “While practitioners often reduce costs by using fewer samples (Perlitz et al., 2024; Polo et al., 2024a; Fogliato et al., 2024; Zhao et al., 2024), these heuristic approaches lack statistical guarantees.”

    From Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data (2026) · cites this paper as Fogliato et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 67 · Cited by 2

Research works printed but not linked: 98

  1. Angelopoulos, A.N., Bates, S., Fannjiang, C., Jordan, M.I., Zrnic, T.: Prediction-powered inference. Science 382(6671), 669-674 (2023)
  2. Angelopoulos, A.N., Duchi, J.C., Zrnic, T.: Ppi++: Efficient prediction-powered inference. arXiv preprint arXiv:2311.01453 (2023) · arXiv
  3. Ash, J.T., Zhang, C., Krishnamurthy, A., Langford, J., Agarwal, A.: Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671 (2019) · arXiv
  4. Baek, C., Jiang, Y., Raghunathan, A., Kolter, J.Z.: Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems 35, 19274-19289 (2022)
  5. Barbu, A., Mayo, D., Alverio, J., Luo, W., Wang, C., Gutfreund, D., Tenenbaum, J., Katz, B.: Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems 32 (2019)
  6. Beery, S., Cole, E., Gjoka, A.: The iwildcam 2020 competition dataset. arXiv preprint arXiv:2004.10340 (2020) · arXiv
  7. Breidt, F.J., Claeskens, G., Opsomer, J.: Model-assisted estimation for complex surveys using penalised splines. Biometrika 92(4), 831-846 (2005)
  8. Breidt, F.J., Opsomer, J.D.: Model-Assisted Survey Estimation with Modern Prediction Techniques. Statistical Science 32(2), 190 -205 (2017). https://doi.org/10.1214/16-STS589 · DOI
  9. Brus, D.J.: Spatial sampling with R. CRC Press (2022)
  10. Chen, M., Goel, K., Sohoni, N.S., Poms, F., Fatahalian, K., Ré, C.: Mandoline: Model evaluation under distribution shift. In: International conference on machine learning. pp. 1617-1629. PMLR (2021)
  11. Chen, T., Lumley, T.: Optimal multiwave sampling for regression modeling in two-phase designs. Statistics in medicine 39(30), 4912-4921 (2020)
  12. Chen, T., Lumley, T.: Optimal sampling for design-based estimators of regression models. Statistics in medicine 41(8), 1482-1497 (2022)
  13. Chen, Y., Zhang, S., Song, R.: Scoring your prediction on unseen data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. pp. 3279-3288 (June 2023)
  14. Cheng, G., Han, J., Lu, X.: Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE 105(10), 1865-1883 (2017)
  15. Chouldechova, A., Deng, S., Wang, Y., Xia, W., Perona, P.: Unsupervised and semi-supervised bias benchmarking in face recognition. In: European Conference on Computer Vision. pp. 289-306. Springer (2022)
  16. Chu, W., Zinkevich, M., Li, L., Thomas, A., Tseng, B.: Unbiased online active learning in data streams. In: Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 195-203 (2011)
  17. Chuang, C.Y., Torralba, A., Jegelka, S.: Estimating generalization under distribution shifts via domain-invariant representations. arXiv preprint arXiv:2007.03511 (2020) · arXiv
  18. Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., , Vedaldi, A.: Describing textures in the wild. In: Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) (2014)
  19. Clark, R.G., Steel, D.G.: Sample design for analysis using high-influence probability sampling. Journal of the Royal Statistical Society Series A: Statistics in Society 185(4), 1733-1756 (2022)
  20. Coates, A., Ng, A., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: Proceedings of the fourteenth international conference on artificial intelligence and statistics. pp. 215-223. JMLR Workshop and Conference Proceedings (2011)
  21. Cochran, W.G.: Sampling Techniques. John Wiley & Sons (1977)
  22. Cohn, D.A., Ghahramani, Z., Jordan, M.I.: Active learning with statistical models. Journal of artificial intelligence research 4, 129-145 (1996)
  23. Deng, L.: The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine 29(6), 141-142 (2012)
  24. Deng, W., Gould, S., Zheng, L.: What does rotation prediction tell us about classifier accuracy under varying testing environments? In: International Conference on Machine Learning. pp. 2579-2589. PMLR (2021)
  25. Deng, W., Zheng, L.: Are labels always necessary for classifier accuracy evaluation? In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 15069-15078 (2021)
  26. Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PAS- CAL Visual Object Classes Challenge 2007 (VOC2007) Results. http://www.pascal- network.org/challenges/VOC/voc2007/workshop/index.html
  27. Farquhar, S., Gal, Y., Rainforth, T.: On statistical bias in active learning: How and when to fix it. arXiv preprint arXiv:2101.11665 (2021) · arXiv
  28. Fei-Fei, L., Fergus, R., Perona, P.: Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In: 2004 conference on computer vision and pattern recognition workshop. pp. 178-178. IEEE (2004)
  29. Fuller, W.A.: Sampling Statistics. John Wiley & Sons (2011)
  30. Gal, Y., Islam, R., Ghahramani, Z.: Deep bayesian active learning with image data. In: International conference on machine learning. pp. 1183-1192. PMLR (2017)
  31. Ganti, R., Gray, A.: Upal: Unbiased pool based active learning. In: Artificial Intelligence and Statistics. pp. 422-431. PMLR (2012)
  32. Garg, S., Balakrishnan, S., Lipton, Z.C., Neyshabur, B., Sedghi, H.: Leveraging unlabeled data to predict out-of-distribution performance. arXiv preprint arXiv:2201.04234 (2022) · arXiv
  33. Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: The kitti dataset. Interna- tional Journal of Robotics Research (IJRR) (2013)
  34. Graubardand, B.I., Korn, E.L.: Inference for superpopulation parameters using sample surveys. Statistical Science 17(1), 73-96 (2002)
  35. Guillory, D., Shankar, V., Ebrahimi, S., Darrell, T., Schmidt, L.: Predicting with confidence on unseen distributions. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1134-1144 (2021)
  36. Hájek, J.: Optimal strategy and other problems in probability sampling. Časopis pro pěstování matematiky 84(4), 387-423 (1959)
  37. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778 (2016)
  38. Hébert-Johnson, U., Kim, M., Reingold, O., Rothblum, G.: Multicalibration: Calibration for the (computationally-identifiable) masses. In: International Conference on Machine Learning. pp. 1939-1948. PMLR (2018)
  39. Helber, P., Bischke, B., Dengel, A., Borth, D.: Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (2019)
  40. Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., Gilmer, J.: The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV (2021)
  41. Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., Song, D.: Natural adversarial examples. CVPR (2021)
  42. Horvitz, D.G., Thompson, D.J.: A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association 47(260), 663-685 (1952)
  43. Imberg, H., Axelson-Fisk, M., Jonasson, J.: Optimal subsampling designs. arXiv preprint arXiv:2304.03019 (2023) · arXiv
  44. Imberg, H., Jonasson, J., Axelson-Fisk, M.: Optimal sampling in unbiased active learning. In: International Conference on Artificial Intelligence and Statistics. pp. 559-569. PMLR (2020)
  45. Imberg, H., Yang, X., Flannagan, C., Bärgman, J.: Active sampling: A machine-learning- assisted framework for finite population inference with optimal subsamples. arXiv preprint arXiv:2212.10024 (2022) · arXiv
  46. Isaki, C.T., Fuller, W.A.: Survey design under the regression superpopulation model. Journal of the American Statistical Association 77(377), 89-96 (1982)
  47. Jiang, Y., Nagarajan, V., Baek, C., Kolter, J.Z.: Assessing generalization of sgd via disagree- ment. arXiv preprint arXiv:2106.13799 (2021) · arXiv
  48. Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., Girshick, R.: Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2901-2910 (2017)
  49. Kim, M.P., Kern, C., Goldwasser, S., Kreuter, F., Reingold, O.: Universal adaptability: Target- independent inference that competes with propensity scoring. Proceedings of the National Academy of Sciences 119(4), e2108097119 (2022)
  50. Kirsch, A., Van Amersfoort, J., Gal, Y.: Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems 32 (2019)
  51. Koh, P.W., Sagawa, S., Marklund, H., Xie, S.M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R.L., Gao, I., et al.: Wilds: A benchmark of in-the-wild distribution shifts. In: International Conference on Machine Learning. pp. 5637-5664. PMLR (2021)
  52. Kossen, J., Farquhar, S., Gal, Y., Rainforth, T.: Active surrogate estimators: An active learning approach to label-efficient model evaluation. In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (eds.) Advances in Neural Information Processing Systems. vol. 35, pp. 24557-24570.
  53. Kossen, J., Farquhar, S., Gal, Y., Rainforth, T.: Active testing: Sample-efficient model evaluation. In: International Conference on Machine Learning. pp. 5753-5763. PMLR (2021)
  54. Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3d object representations for fine-grained cat- egorization. In: 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13). Sydney, Australia (2013)
  55. Kull, M., Flach, P.: Novel decompositions of proper scoring rules for classification: Score adjustment as precursor to calibration. In: Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2015, Porto, Portugal, September 7-11, 2015, Proceedings, Part I 15. pp. 68-85. Springer (2015)
  56. LeCun, Y., Huang, F.J., Bottou, L.: Learning methods for generic object recognition with invariance to pose and lighting. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. vol. 2, pp. II-104. IEEE (2004)
  57. Lewis, D.D.: A sequential algorithm for training text classifiers: Corrigendum and additional data. In: Acm Sigir Forum. vol. 29, pp. 13-19. ACM New York, NY, USA (1995)
  58. Lewis, D.D., Catlett, J.: Heterogeneous uncertainty sampling for supervised learning. In: Machine learning proceedings 1994, pp. 148-156. Elsevier (1994)
  59. Li, Z., Ma, X., Xu, C., Cao, C., Xu, J., Lü, J.: Boosting operational dnn testing efficiency through conditioning (2019). https://doi.org/10.1145/3338906.3338930 · DOI
  60. Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 11976-11986 (2022)
  61. Lohr, S.L.: Sampling: design and analysis. CRC press (2021)
  62. Lumley, T., Shaw, P.A., Dai, J.Y.: Connections between survey calibration estimators and semiparametric models for incomplete data. International Statistical Review 79(2), 200-220 (2011)
  63. Maji, S., Rahtu, E., Kannala, J., Blaschko, M., Vedaldi, A.: Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 (2013) · arXiv
  64. McConville, K.S., Breidt, F.J., Lee, T.C., Moisen, G.G.: Model-assisted survey regression estimation with the lasso. Journal of Survey Statistics and Methodology 5(2), 131-158 (2017)
  65. Miller, B.A., Vila, J., Kirn, M., Zipkin, J.R.: Classifier performance estimation with unbalanced, partially labeled data. In: Torgo, L., Matwin, S., Weiss, G., Moniz, N., Branco, P. (eds.) Proceedings of The International Workshop on Cost-Sensitive Learning. Proceedings of Machine Learning Research, vol. 88, pp. 4-16. PMLR (05 May 2018)
  66. Miller, J.P., Taori, R., Raghunathan, A., Sagawa, S., Koh, P.W., Shankar, V., Liang, P., Carmon, Y., Schmidt, L.: Accuracy on the line: on the strong correlation between out- of-distribution and in-distribution generalization. In: International Conference on Machine Learning. pp. 7721-7735. PMLR (2021)
  67. Neyman, J.: On the two different aspects of the representative method: the method of stratified sampling and the method of purposive selection. In: Breakthroughs in Statistics: Methodology and Distribution, pp. 123-150. Springer (1992)
  68. Nilsback, M.E., Zisserman, A.: Automated flower classification over a large number of classes. In: 2008 Sixth Indian conference on computer vision, graphics & image processing. pp. 722-729. IEEE (2008)
  69. Parkhi, O.M., Vedaldi, A., Zisserman, A., Jawahar, C.: Cats and dogs. In: 2012 IEEE conference on computer vision and pattern recognition. pp. 3498-3505. IEEE (2012)
  70. Poms, F., Sarukkai, V., Mullapudi, R.T., Sohoni, N.S., Mark, W.R., Ramanan, D., Fatahalian, K.: Low-shot validation: Active importance sampling for estimating classifier performance on rare categories. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 10705-10714 (October 2021)
  71. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748-8763. PMLR (2021)
  72. Recht, B., Roelofs, R., Schmidt, L., Shankar, V.: Do imagenet classifiers generalize to imagenet? In: International conference on machine learning. pp. 5389-5400. PMLR (2019)
  73. Ren, P., Xiao, Y., Chang, X., Huang, P.Y., Li, Z., Gupta, B.B., Chen, X., Wang, X.: A survey of deep active learning. ACM computing surveys (CSUR) 54(9), 1-40 (2021)
  74. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recog- nition Challenge. International Journal of Computer Vision (IJCV) 115(3), 211-252 (2015). https://doi.org/10.1007/s11263-015-0816-y · DOI · Paper page
  75. Särndal, C.E.: The calibration approach in survey theory and practice. Survey methodology 33(2), 99-119 (2007)
  76. Särndal, C.E., Swensson, B., Wretman, J.: Model assisted survey sampling. Springer Science & Business Media (2003)
  77. Sawade, C., Landwehr, N., Bickel, S., Scheffer, T.: Active risk estimation. In: Proceedings of the 27th International Conference on International Conference on Machine Learning. p. 951-958. ICML'10, Omnipress, Madison, WI, USA (2010)
  78. Sawade, C., Landwehr, N., Scheffer, T.: Active estimation of f-measures. In: Lafferty, J., Williams, C., Shawe-Taylor, J., Zemel, R., Culotta, A. (eds.) Advances in Neural Information Processing Systems. vol. 23. Curran Associates, Inc. (2010)
  79. Scheffer, T., Decomain, C., Wrobel, S.: Active hidden markov models for information extraction. In: International symposium on intelligent data analysis. pp. 309-318. Springer (2001)
  80. Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C.W., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., Schramowski, P., Kundurthy, S.R., Crowson, K., Schmidt, L., Kaczmarczyk, R., Jitsev, J.: LAION-5b: An open large-scale dataset for training next generation image-text models. In: Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2022), https: //openreview.net/forum?id=M3Y74vmsMcY
  81. Sener, O., Savarese, S.: Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 (2017) · arXiv
  82. Siddhant, A., Lipton, Z.C.: Deep bayesian active learning for natural language processing: Results of a large-scale empirical study. arXiv preprint arXiv:1808.05697 (2018) · arXiv
  83. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 conference on empirical methods in natural language processing. pp. 1631-1642 (2013)
  84. Stallkamp, J., Schlipsing, M., Salmen, J., Igel, C.: The german traffic sign recognition bench- mark: a multi-class classification competition. In: The 2011 international joint conference on neural networks. pp. 1453-1460. IEEE (2011)
  85. Taylor, J., Earnshaw, B., Mabey, B., Victors, M., Yosinski, J.: Rxrx1: An image set for cellular morphological variation across many experimental batches. In: International Conference on Learning Representations (ICLR) (2019)
  86. Tillé, Y.: Sampling and estimation from finite populations. John Wiley & Sons (2020)
  87. Veeling, B.S., Linmans, J., Winkens, J., Cohen, T., Welling, M.: Rotation equivariant cnns for digital pathology. In: Medical Image Computing and Computer Assisted Intervention-MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11. pp. 210-218. Springer (2018)
  88. Wald, Y., Feder, A., Greenfeld, D., Shalit, U.: On calibration and out-of-domain generalization. Advances in neural information processing systems 34, 2215-2227 (2021)
  89. Wang, H., Ge, S., Lipton, Z., Xing, E.P.: Learning robust global representations by penalizing local predictive power. In: Advances in Neural Information Processing Systems. pp. 10506- 10518 (2019)
  90. Welinder, P., Welling, M., Perona, P.: A lazy man's approach to benchmarking: Semisupervised classifier evaluation and recalibration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2013)
  91. Wenzel, F., Dittadi, A., Gehler, P., Simon-Gabriel, C.J., Horn, M., Zietlow, D., Kernert, D., Russell, C., Brox, T., Schiele, B., et al.: Assaying out-of-distribution generalization in transfer learning. Advances in Neural Information Processing Systems 35, 7181-7198 (2022)
  92. Wu, C., Sitter, R.R.: A model-calibration approach to using complete auxiliary information from survey data. Journal of the American Statistical Association 96(453), 185-193 (2001)
  93. Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: Large-scale scene recog- nition from abbey to zoo. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. pp. 3485-3492 (June 2010). https://doi.org/10.1109/CVPR.2010.5539970 · DOI · Paper page
  94. Yu, Y., Bates, S., Ma, Y., Jordan, M.: Robust calibration with multi-domain temperature scaling. Advances in Neural Information Processing Systems 35, 27510-27523 (2022)
  95. Yu, Y., Yang, Z., Wei, A., Ma, Y., Steinhardt, J.: Predicting out-of-distribution error with the projection norm. In: Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., Sabato, S. (eds.) Proceedings of the 39th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 162, pp. 25721-25746. PMLR (17-23 Jul 2022)
  96. Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M., Djolonga, J., Pinto, A.S., Neumann, M., Dosovitskiy, A., Beyer, L., Bachem, O., Tschannen, M., Michalski, M., Bousquet, O., Gelly, S., Houlsby, N.: The visual task adaptation benchmark (2020), https://openreview.net/forum?id=BJena3VtwS
  97. Zrnic, T., Candès, E.J.: Active statistical inference. arXiv preprint arXiv:2403.03208 (2024) · arXiv
  98. Zrnic, T., Candès, E.J.: Cross-prediction-powered inference. Proceedings of the National Academy of Sciences 121(15), e2322083121 (2024)

Authors
Fogliato, R., Patil, P., Monfort, M., & Perona, P.
Affiliations
  • Amazon (United States) (US)
  • University of California, Berkeley (US)
  • California Institute of Technology (US)
Corresponding author
Fogliato, Riccardo
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Pages
140-158
ISSN
0302-9743 (ISSN-L)
DOI
10.48550/arxiv.2406.07320
arXiv
2406.07320
OpenAlex
W4404125036
Cited by
2
References
67
Citations by year
2025: 2
FWCI
0.58
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Lecture notes in computer science (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
  • Physical Sciences › Mathematics › Statistics and Probability › Survey Sampling and Estimation Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Imbalanced Data Classification Techniques
Keywords
model evaluation, stratified sampling, K-means clustering, accuracy estimation, model-assisted estimation, annotation cost reduction, evaluation efficiency, computer vision datasets, statistical estimation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review