Natural Adversarial Examples
Open accessAbstract
We introduce two challenging datasets that reliably cause machine learning model performance to substantially degrade. The datasets are collected with a simple adversarial filtration technique to create datasets with limited spurious cues. Our datasets’ real-world, unmodified examples transfer to various unseen models reliably, demonstrating that computer vision models have shared weaknesses. The first dataset is called IMAGENET-A and is like the ImageNet test set, but it is far more challenging for existing models. We also curate an adversarial out-of-distribution detection dataset called IMAGENET-O, which is the first out-of-distribution detection dataset created for ImageNet models. On IMAGENET-A a DenseNet-121 obtains around 2% accuracy, an accuracy drop of approximately 90%, and its out-of-distribution detection performance on IMAGENET-O is near random chance levels. We find that existing data augmentation techniques hardly boost performance, and using other public training datasets provides improvements that are limited. However, we find that improvements to computer vision architectures provide a promising path towards robust models.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
MEMO is the first adaptation method to improve performance, by 1-4% over standard model evaluation, on the ImageNet-A test set.
“Also, MEMO is, to the best of our knowledge, the first adaptation method to improve performance (by 1-4% over standard model evaluation) on the ImageNet-A test set [15].”
From MEMO: Test Time Robustness Via Adaptation and Augmentation (2022) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Part of this paper’s reference list could not be split into single entries.
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · see citing works
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017) · arXiv · doi:10.48550/arxiv.1706.08500 · Cited by 8,365 · see citing works
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309 · see citing works
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196 · see citing works
Self-Attention Generative Adversarial Networks
Zhang, H., Goodfellow, I., Metaxas, D., & Odena, A. (2018) · arXiv · doi:10.48550/arxiv.1805.08318 · Cited by 3,353 · see citing works
Describing Textures in the Wild
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., & Vedaldi, A. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.461 · Cited by 2,438 · see citing works
Adversarial machine learning at scale
Kurakin, A., Goodfellow, I., & Bengio, S. (2016) · arXiv · doi:10.48550/arxiv.1611.01236 · Cited by 2,325 · see citing works
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224 · see citing works
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
Hendrycks, D., & Dietterich, T. (2019) · arXiv · doi:10.48550/arxiv.1903.12261 · Cited by 1,335 · see citing works
Self-citationHellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830 · see citing works
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1246 · Cited by 393 · cited 1× · see citing works
“Many recently introduced NLP datasets use adversarial filtration to create “adversarial datasets” by sieving examples solved with simple spurious cues [44, 4, 56, 13, 6, 25].”
Background
Flagged first, then by citations
Research works printed but not linked: 47
- Faruk Ahmed and Aaron C. Courville. Detecting semantic anomalies. ArXiv, abs/1908.04388, 2019. · arXiv
- Martín Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. ArXiv, abs/1907.02893, 2019. · arXiv
- David Bau, B. Zhou, A. Khosla, A. Oliva, and A. Torralba. Network dissection: Quantifying interpretability of deep vi- sual representations. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3319-3327, 2017.
- Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Yih, and Yejin Choi. Abductive common- sense reasoning. ArXiv, abs/1908.05739, 2019. · arXiv
- Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical common- sense in natural language. ArXiv, abs/1911.11641, 2019. · arXiv
- Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical common- sense in natural language. ArXiv, abs/1911.11641, 2020. · arXiv
- Wieland Brendel and Matthias Bethge. Approximating cnns with bag-of-local-features models works surprisingly well on imagenet. CoRR, abs/1904.00760, 2018. · arXiv
- Zheng Cai, Lifu Tu, and Kevin Gimpel. Pay attention to the ending: Strong neural baselines for the roc story cloze task. In ACL, 2017.
- Jia Deng, Wei Dong, Richard Socher, Li jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. CVPR, 2009.
- Terrance Devries and Graham W. Taylor. Improved regular- ization of convolutional neural networks with Cutout. arXiv preprint arXiv:1708.04552, 2017. · arXiv
- Terrance Devries and Graham W. Taylor. Learning confi- dence for out-of-distribution detection in neural networks. ArXiv, abs/1802.04865, 2018. · arXiv
- A. Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, M. De- hghani, Matthias Minderer, Georg Heigold, S. Gelly, Jakob Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
- L. Engstrom, Andrew Ilyas, Shibani Santurkar, D. Tsipras, J. Steinhardt, and A. Madry. Identifying statistical bias in dataset replication. ArXiv, abs/2005.09619, 2020. · arXiv
- Shanghua Gao, Ming-Ming Cheng, Kai Zhao, Xinyu Zhang, Ming-Hsuan Yang, and Philip H. S. Torr. Res2net: A new multi-scale backbone architecture. IEEE transactions on pattern analysis and machine intelligence, 2019.
- Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. ICLR, 2019.
- Robert Geirhos, Carlos R. M. Temme, Jonas Rauber, Heiko H. Schütt, Matthias Bethge, and Felix A. Wich- mann. Generalisation in humans and deep neural networks. NeurIPS, 2018.
- Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. An- notation artifacts in natural language inference data. ArXiv, abs/1803.02324, 2018. · arXiv
- Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask r-cnn. In CVPR, 2018.
- Dan Hendrycks, Steven Basart, Mantas Mazeika, Moham- madreza Mostajabi, J. Steinhardt, and D. Song. Scaling out-of-distribution detection for real-world settings. arXiv: 1911.11132, 2020. · arXiv
- Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kada- vath, F. Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli, M. Guo, D. Song, J. Steinhardt, and J. Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ArXiv, abs/2006.16241, 2020. · arXiv
- Dan Hendrycks, C. Burns, Steven Basart, Andrew Critch, Jerry Li, D. Song, and J. Steinhardt. Aligning ai with shared human values. ArXiv, abs/2008.02275, 2020. · arXiv
- Dan Hendrycks and Thomas Dietterich. Benchmarking neu- ral network robustness to common corruptions and perturba- tions. ICLR, 2019.
- Dan Hendrycks and Kevin Gimpel. A baseline for detect- ing misclassified and out-of-distribution examples in neural networks. ICLR, 2017.
- Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. ICLR, 2019.
- Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances in Neural In- formation Processing Systems (NeurIPS), 2019.
- Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and D. Song. Using self-supervised learning can improve model ro- bustness and uncertainty. In NeurIPS, 2019.
- Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. ICLR, 2020.
- Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Andrea Vedaldi. Gather-excite : Exploiting feature context in convo- lutional neural networks. In NeurIPS, 2018.
- Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitation net- works. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
- Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara Balan, Alireza Fathi, Ian Fischer, Zbig- niew Wojna, Yang Song, Sergio Guadarrama, and Kevin Murphy. Speed/accuracy trade-offs for modern convolu- tional object detectors. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
- Simon Kornblith, Jonathon Shlens, and Quoc V. Le. Do bet- ter imagenet models transfer better? CoRR, abs/1805.08974, 2018. · arXiv
- A. Kumar, P. Liang, and T. Ma. Verified uncertainty calibra- tion. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
- Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out- of-distribution samples. ICLR, 2018.
- Bo-Yi Li, Felix Wu, Ser-Nam Lim, Serge J. Belongie, and Kilian Q. Weinberger. On feature normalization and data augmentation. ArXiv, abs/2002.11102, 2020. · arXiv
- Alexander Meinke and Matthias Hein. Towards neural net- works that provably know when they don't know. ArXiv, abs/1909.12180, 2019. · arXiv
- Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to im- agenet? ArXiv, abs/1902.10811, 2019. · arXiv
- Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. ArXiv, abs/1907.10641, 2019. · arXiv
- Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Ba- tra. Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Com- puter Vision, 128:336 -359, 2019.
- Pierre Stock and Moustapha Cissé. Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases. In ECCV, 2018.
- D. Su, Huan Zhang, H. Chen, Jinfeng Yi, P. Chen, and Yu- peng Gao. Is robustness the cost of accuracy? -a comprehen- sive study on the robustness of 18 deep image classification models. In ECCV, 2018.
- Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers and distillation through at- tention. arXiv preprint arXiv:2012.12877, 2020. · arXiv
- Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994, 2020. · arXiv
- Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. 2010 IEEE Computer Soci- ety Conference on Computer Vision and Pattern Recognition, pages 3485-3492, 2010.
- Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. CVPR, 2016.
- Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, E. Cubuk, and J. Gilmer. A fourier perspective on model ro- bustness in computer vision. ArXiv, abs/1906.08988, 2019. · arXiv
- Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regu- larization strategy to train strong classifiers with localizable features. 2019 IEEE/CVF International Conference on Com- puter Vision (ICCV), pages 6022-6031, 2019.
- Hongyi Zhang, Moustapha Cissé, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ArXiv, abs/1710.09412, 2018. · arXiv
Cited by
Sorted by citations
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., & Xie, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01167 · Cited by 8,494 · see citing works
Masked Autoencoders Are Scalable Vision Learners
He, K., Chen, X., Xie, S., Li, Y., Dollar, P., & Girshick, R. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01553 · Cited by 8,489 · see citing works
Learning to Prompt for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01653-1 · Cited by 3,304 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224 · see citing works
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., & Duerig, T. (2021) · arXiv · doi:10.48550/arxiv.2102.05918 · Cited by 1,977 · see citing works
“…ILSVRC-2012 benchmark (Deng et al., 2009) and its variants including ImageNet-R(endition) (Hendrycks et al., 2020) (non-natural images such as art, cartoons, sketches), ImageNet-A(dversarial) (Hendrycks et al., 2021) (more challenging images for ML models), and ImageNet-V2 (Recht et al., 2019).”
BackgroundDINOv2: Learning Robust Visual Features without Supervision
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.07193 · Cited by 1,078 · see citing works
“1% on R (Hendrycks et al., 2021a) and +23 .”
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02283 · Cited by 734 · see citing works
Understanding Robustness of Transformers for Image Classification
Bhojanapalli, S., Chakrabarti, A., Glasner, D., Li, D., Unterthiner, T., & Veit, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01007 · Cited by 440 · see citing works
Robust fine-tuning of zero-shot models
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00780 · Cited by 428 · see citing works
MultiMAE: Multi-modal Multi-task Masked Autoencoders
Bachmann, R., Mizrahi, D., Atanov, A., & Zamir, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_20 · Cited by 268 · see citing works
Vision Transformers Are Robust Learners
Paul, S., & Chen, P.-Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i2.20103 · Cited by 213 · see citing works
“Attention has been shown to be an important element for vision networks to achieve better empirical robustness [9].”
BackgroundMethodInfluentialSymbolic Discovery of Optimization Algorithms
Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.06675 · Cited by 303 · see citing works
DeepNet: Scaling Transformers to 1,000 Layers
Wang, H., Ma, S., Dong, L., Huang, S., Zhang, D., & Wei, F. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3386927 · Cited by 113 · see citing works
Deep Deterministic Uncertainty: A New Simple Baseline
Mukhoti, J., Kirsch, A., van Amersfoort, J., Torr, P. H. S., & Gal, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02336 · Cited by 116 · see citing works
Measure and Improve Robustness in NLP Models: A Survey
Wang, X., Wang, H., & Yang, D. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.339 · Cited by 90 · see citing works
Miller, J., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., Liang, P., Carmon, Y., & Schmidt, L. (2021) · arXiv · doi:10.48550/arxiv.2107.04649 · Cited by 74 · see citing works
MEMO: Test Time Robustness Via Adaptation and Augmentation
Zhang, M., Levine, S., & Finn, C. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2799 · Cited by 189 · see citing works
Generating High Fidelity Data from Low-density Regions using Diffusion Models
Sehwag, V., Hazirbas, C., Gordo, A., Ozgenel, F., & Ferrer, C. C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01120 · Cited by 41 · see citing works
TinySpeech: Attention Condensers for Deep Speech Recognition Neural Networks on Edge Devices
Wong, A., Famouri, M., Pavlova, M., & Surana, S. (2020) · arXiv · doi:10.48550/arxiv.2008.04245 · Cited by 27 · see citing works
- Authors
- Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D.
- Affiliations
- University of California, Berkeley (US)
- University of Washington (US)
- University of Chicago (US)
- Venue
- 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2021-06
- Pages
- 15257-15266
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W3177096435
- Cited by
- 1,090
- References
- 13
- Citations by year
- 2026: 135 · 2025: 252 · 2024: 203 · 2023: 196 · 2022: 102 · 2021: 119 · 2020: 73 · 2019: 10
- FWCI
- 69.04
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- adversarial examples, transferability, out-of-distribution detection, ImageNet, data augmentation, model weaknesses, DenseNet121
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
