CoChat
Sign Up
PreprintCited by 398Domain Adaptation and Few-Shot LearningAdversarial Robustness in Machine LearningMachine Learning and Data Classification

Underspecification Presents Challenges for Credibility in Modern Machine Learning

Open access
D'Amour, Alexander, Heller, Katherine, Moldovan, Dan, Adlam, Ben, Alipanahi, Babak, Beutel, Alex, Chen, Christina, Deaton, Jonathan +32 more·2020·arXiv
DOI 10.48550/arxiv.2011.03395arXiv 2011.03395
Open this paper in CoChat — free accountOpen PDF

Abstract

ML models often exhibit unexpectedly poor behavior when they are deployed in real-world domains. We identify underspecification as a key reason for these failures. An ML pipeline is underspecified when it can return many predictors with equivalently strong held-out performance in the training domain. Underspecification is common in modern ML pipelines, such as those based on deep learning. Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very differently in deployment domains. This ambiguity can lead to instability and poor model behavior in practice, and is a distinct failure mode from previously identified issues arising from structural mismatch between training and deployment domains. We show that this problem appears in a wide variety of practical ML pipelines, using examples from computer vision, medical imaging, natural language processing, clinical risk prediction based on electronic health records, and medical genomics. Our results show the need to explicitly account for underspecification in modeling pipelines that are intended for real-world deployment in any domain.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2011.03395

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 107

  1. Ademide Adelekun, Ginikanwa Onyekaba, and Jules B Lipoff. Skin color in dermatology textbooks: An updated evaluation and analysis. Journal of the American Academy of Dermatology , 2020.
  2. R Ambrosino, B G Buchanan, G F Cooper, and M J Fine. The use of misclassification costs to learn rule-based decision support models for cost-effective hospital admission strategies. Proceedings. Symposium on Computer Applications in Medical Care , pages 304-8, 1995. ISSN 01954210. URL http://www.ncbi.nlm.nih.gov/pubmed/8563290http://www.pubmedcentral.nih. gov/articlerender.fcgi?artid=PMC2579104 .
  3. Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019. · arXiv
  4. Susan Athey. Beyond prediction: Using big data for policy problems. Science , 355(6324):483-485, 2017.
  5. Marzieh Babaeianjelodar, Stephen Lorenz, Josh Gordon, Jeanna Matthews, and Evan Freitag. Quantifying gender bias in different corpora. In Companion Proceedings of the Web Conference 2020 , pages 752-759, 2020.
  6. Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In Advances in Neural Information Processing Systems , pages 9448-9458, 2019.
  7. Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine learning and the bias-variance trade-off. arXiv preprint arXiv:1812.11118 , 2018. · arXiv
  8. Yoshua Bengio. The consciousness prior. arXiv preprint arXiv:1709.08568 , 2017. · arXiv
  9. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 632-642, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1075. URL https: //www.aclweb.org/anthology/D15-1075 . · DOI · Paper page
  10. Kendrick Boyd, Vítor Santos Costa, Jesse Davis, and C. David Page. Unachievable region in precision-recall space and its effect on empirical evaluation. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012 , volume 1, pages 639-646, 2012. ISBN 9781450312851.
  11. Theodora S. Brisimi, Tingting Xu, Taiyao Wang, Wuyang Dai, and Ioannis Ch Paschalidis. Predicting diabetes-related hospitalizations based on electronic health records. Statistical Methods in Medical Research , 28(12):3667-3682, dec 2019. ISSN 14770334. doi: 10.1177/0962280218810911. · DOI
  12. Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency , pages 77-91, 2018.
  13. Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. Semantics derived automatically from language corpora contain human-like biases. Science , 356(6334):183-186, 2017.
  14. CARDIoGRAMplusC4D Consortium, Panos Deloukas, Stavroula Kanoni, Christina Willenborg, Martin Farrall, Themistocles L Assimes, John R Thompson, Erik Ingelsson, Danish Saleheen, Jeanette Erdmann, Benjamin A Goldstein, Kathleen Stirrups, Inke R König, Jean-Baptiste Cazier, Asa Johansson, Alistair S Hall, Jong-Young Lee, Cristen J Willer, John C Chambers, Tõnu Esko, Lasse Folkersen, Anuj Goel, Elin Grundberg, Aki S Havulinna, Weang K Ho, Jemma C Hopewell, Niclas Eriksson, Marcus E Kleber, Kati Kristiansson, Per Lundmark, Leo-Pekka Lyytikäinen, Suzanne Rafelt, Dmitry Shungin, Rona J Strawbridge, Gudmar Thorleifsson, Emmi Tikkanen, Natalie Van Zuydam, Benjamin F Voight, Lindsay L Waite, Weihua Zhang, Andreas Ziegler, Devin Absher, David Altshuler, Anthony J Balmforth, Inês Barroso, Peter S Braund, Christof Burgdorf, Simone Claudi-Boehm, David Cox, Maria Dimitriou, Ron Do, DIAGRAM Consortium, CARDIOGENICS Consortium, Alex S F Doney, Noureddine El Mokhtari, Per Eriksson, Krista Fischer, Pierre Fontanillas, Anders Franco-Cereceda, Bruna Gigante, Leif Groop, Stefan Gustafsson, Jörg Hager, Göran Hallmans, Bok-Ghee Han, Sarah E Hunt, Hyun M Kang, Thomas Illig, Thorsten Kessler, Joshua W Knowles, Genovefa Kolovou, Johanna Kuusisto, Claudia Langenberg, Cordelia Langford, Karin Leander, Marja-Liisa Lokki, Anders Lundmark, Mark I McCarthy, Christa Meisinger, Olle Melander, Evelin Mihailov, Seraya Maouche, Andrew D Morris, Martina Müller-Nurasyid, MuTHER Consortium, Kjell Nikus, John F Peden, N William Rayner, Asif Rasheed, Silke Rosinger, Diana Rubin, Moritz P Rumpf, Arne Schäfer, Mohan Sivananthan, Ci Song, Alexandre F R Stewart, Sian-Tsung Tan, Gudmundur Thorgeirsson, C Ellen van der Schoot, Peter J Wagner, Wellcome Trust Case Control Consortium, George A Wells, Philipp S Wild, Tsun-Po Yang, Philippe Amouyel, Dominique Arveiler, Hanneke Basart, Michael Boehnke, Eric Boerwinkle, Paolo Brambilla, Francois Cambien, Adrienne L Cupples, Ulf de Faire, Abbas Dehghan, Patrick Diemert, Stephen E Epstein, Alun Evans, Marco M Ferrario, Jean Ferrières, Dominique Gauguier, Alan S Go, Alison H Goodall, Villi Gudnason, Stanley L Hazen, Hilma Holm, Carlos Iribarren, Yangsoo Jang, Mika Kähönen, Frank Kee, Hyo-Soo Kim, Norman Klopp, Wolfgang Koenig, Wolfgang Kratzer, Kari Kuulasmaa, Markku Laakso, Reijo Laaksonen, JiYoung Lee, Lars Lind, Willem H Ouwehand, Sarah Parish, Jeong E Park, Nancy L Pedersen, Annette Peters, Thomas Quertermous, Daniel J Rader, Veikko Salomaa, Eric Schadt, Svati H Shah, Juha Sinisalo, Klaus Stark, Kari Stefansson, David-Alexandre Trégouët, Jarmo Virtamo, Lars Wallentin, Nicholas Wareham, Martina E Zimmermann, Markku S Nieminen, Christian Hengstenberg, Manjinder S Sandhu, Tomi Pastinen, Ann-Christine Syvänen, G Kees Hovingh, George Dedoussis, Paul W Franks, Terho Lehtimäki, Andres Metspalu, Pierre A Zalloua, Agneta Siegbahn, Stefan Schreiber, Samuli Ripatti, Stefan S Blankenberg, Markus Perola, Robert Clarke, Bernhard O Boehm, Christopher O'Donnell, Muredach P Reilly, Winfried März, Rory Collins, Sekar Kathiresan, Anders Hamsten, Jaspal S Kooner, Unnur Thorsteinsdottir, John Danesh, Colin N A Palmer, Robert Roberts, Hugh Watkins, Heribert Schunkert, and Nilesh J Samani. Large-scale association analysis identifies new risk loci for coronary artery disease. Nat. Genet. , 45(1):25-33, January 2013.
  15. Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Marc Sturm, and Noemie Elhadad. Intelligible Models for HealthCare. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '15 , pages 1721-1730, 2015. ISBN 9781450336642. doi: 10.1145/2783258.2788613. URL http://dx.doi.org/10.1145/2783258.2788613http://dl. acm.org/citation.cfm?doid=2783258.2788613 . http://dx.doi.org/10.1145/2783258.2788613%20http://dl.acm.org/citation.cfm?doid=2783258.2788613 · DOI · Paper page
  16. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) , pages 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/S17-2001. URL https://www.aclweb.org/anthology/S17-2001 . · DOI · Paper page
  17. Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment , 2019(12):124018, 2019.
  18. Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F Stewart, and Jimeng Sun. GRAM: Graph-based Attention Model for Healthcare Representation Learning. 2017. doi: 10.1145/3097983. 3098126. URL http://dx.doi.org/10.1145/3097983.3098126 . · DOI · Paper page
  19. Gary S Collins, Johannes B Reitsma, Douglas G Altman, and Karel GM Moons. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (tripod): the tripod statement. British Journal of Surgery , 102(3):148-158, 2015.
  20. Jasmine Collins, Jascha Sohl-Dickstein, and David Sussillo. Capacity and trainability in recurrent neural networks. In 5th International Conference on Learning Representations, ICLR 2017 Conference Track Proceedings , 2017.
  21. Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D'Amour, Dan Moldovan, et al. On robustness and transferability of convolutional neural networks. arXiv preprint arXiv:2007.08558 , 2020. · arXiv
  22. Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. arXiv preprint arXiv:2002.06305 , 2020. · arXiv
  23. Michael W Dusenberry, Dustin Tran, Edward Choi, Jonas Kemp, Jeremy Nixon, Ghassen Jerfel, Katherine Heller, and Andrew M Dai. Analyzing the role of model uncertainty for electronic health records. In Proceedings of the ACM Conference on Health, Inference, and Learning , pages 204-213, 2020.
  24. Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun. Dermatologist-level classification of skin cancer with deep neural networks. Nature , 542(7639):115-118, 2017.
  25. Chenchen Feng, David Le, and Allison B. McCoy. Using Electronic Health Records to Identify Adverse Drug Events in Ambulatory Care: A Systematic Review. Applied Clinical Informatics , 10 (1):123-128, 2019. ISSN 18690327. doi: 10.1055/s-0039-1677738. · DOI
  26. Aaron Fisher, Cynthia Rudin, and Francesca Dominici. All models are wrong, but many are useful: Learning a variable's importance by studying an entire class of prediction models simultaneously. Journal of Machine Learning Research , 20(177):1-81, 2019.
  27. TB Fitzpatrick. Sun and skin. Journal de Medecine Esthetique , 2:33-34, 1975.
  28. Seth Flaxman, Swapnil Mishra, Axel Gandy, H Juliette T Unwin, Thomas A Mellan, Helen Coupland, Charles Whittaker, Harrison Zhu, Tresnia Berah, Jeffrey W Eaton, et al. Estimating the effects of non-pharmaceutical interventions on covid-19 in europe. Nature , 584(7820):257-261, 2020.
  29. Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint arXiv:1912.02757 , 2019. · arXiv
  30. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning , 2020.
  31. Joseph Futoma, Morgan Simons, Trishan Panch, Finale Doshi-Velez, and Leo Anthony Celi. The myth of generalisability in clinical research and machine learning in health care. The Lancet Digital Health , 2(9):e489 - e492, 2020. ISSN 2589-7500. doi: https://doi.org/10.1016/S2589-7500(20)30186-2. URL http://www.sciencedirect.com/science/article/pii/S2589750020301862 . · DOI · Paper page
  32. Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H Chi, and Alex Beutel. Counterfactual fairness in text classification through robustness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society , pages 219-226, 2019.
  33. Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems , pages 8789-8798, 2018.
  34. Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning , volume 1. MIT press Cambridge, 2016.
  35. Varun Gulshan, Lily Peng, Marc Coram, Martin C Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. Jama , 316(22):2402-2410, 2016.
  36. Christina Heinze-Deml, Jonas Peters, and Nicolai Meinshausen. Invariant causal prediction for nonlinear models. Journal of Causal Inference , 6(2), 2018.
  37. Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=HJz6tiCqYm .
  38. Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv:1907.07174 , 2019. · arXiv
  39. Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. arXiv preprint arXiv:2006.16241 , 2020. · arXiv
  40. Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation , 9(8):17351780, 1997. URL http://www7.informatik.tu-muenchen.de/{~}hochreithttp://www.idsia. ch/{~}juergen .
  41. Wolfgang Hoffmann, Ute Latza, Sebastian E Baumeister, Martin Brünger, Nina Buttmann-Schweiger, Juliane Hardt, Verena Hoffmann, André Karch, Adrian Richter, Carsten Oliver Schmidt, et al. Guidelines and recommendations for ensuring good epidemiological practice (gep): a guideline developed by the german society for epidemiology. European journal of epidemiology , 34(3):301-317, 2019.
  42. Sara Hooker. The hardware lottery. arXiv preprint arXiv:2009.06489 , 2020. · arXiv
  43. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. OntoNotes: The 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers , pages 57-60, New York City, USA, June 2006. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/N06-2015 .
  44. Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 328-339, 2018.
  45. Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems , pages 125-136, 2019.
  46. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407 , 2018. · arXiv
  47. Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg. Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in ai. arXiv preprint arXiv:2010.07487 , 2020. · arXiv
  48. Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=Sklgs0NFvr .
  49. John A Kellum and Azra Bihorac. Artificial intelligence to predict aki: is it a breakthrough? Nature Reviews Nephrology , pages 1-2, 2019.
  50. Christopher J Kelly, Alan Karthikesalingam, Mustafa Suleyman, Greg Corrado, and Dominic King. Key challenges for delivering clinical impact with artificial intelligence. BMC medicine , 17(1):195, 2019.
  51. Arif Khwaja. KDIGO clinical practice guidelines for acute kidney injury. Nephron - Clinical Practice , 120(4), oct 2012. ISSN 16602110. doi: 10.1159/000339789. · DOI
  52. Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Large scale learning of general visual representations for transfer. arXiv preprint arXiv:1912.11370 , 2019. · arXiv
  53. Jonathan Krause, Varun Gulshan, Ehsan Rahimy, Peter Karth, Kasumi Widner, Greg S Corrado, Lily Peng, and Dale R Webster. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology , 125(8):1264-1272, 2018.
  54. Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Advances in neural information processing systems , pages 4066-4076, 2017.
  55. Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30 , pages 6402-6413. Curran Associates, Inc., 2017. URL http://papers.nips.cc/paper/ 7219-simple-and-scalable-predictive-uncertainty-estimation-using-deep-ensembles. pdf .
  56. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019. · arXiv
  57. Olivier Ledoit and Sandrine Péché. Eigenvectors of some large sample covariance matrix ensembles. Probability Theory and Related Fields , 151(1-2):233-264, 2011.
  58. Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, and Yoav Artzi. Simple recurrent units for highly parallelizable recurrence. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMzhou 2018 , pages 4470-4481. Association for Computational Linguistics, sep 2018. ISBN 9781948087841. doi: 10.18653/v1/d18-1477. URL http://arxiv.org/abs/1709. 02755 . · DOI
  59. Tal Linzen. How can we accelerate progress towards human-like linguistic generalization? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 5210-5217, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. acl-main.465. URL https://www.aclweb.org/anthology/2020.acl-main.465 . · DOI
  60. Yuan Liu, Ayush Jain, Clara Eng, David H Way, Kang Lee, Peggy Bui, Kimberly Kanada, Guilherme de Oliveira Marinho, Jessica Gallegos, Sara Gabriele, et al. A deep learning system for differential diagnosis of skin diseases. Nature Medicine , pages 1-9, 2020b.
  61. Sara Magliacane, Thijs van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31 (NeurIPS2018) , pages 10869-10879. Curran Associates, Inc., 2018. URL http://papers.nips.cc/paper/ 8282-domain-adaptation-by-using-causal-inference-to-predict-invariant-conditional-distributions. pdf .
  62. Alicia R Martin, Masahiro Kanai, Yoichiro Kamatani, Yukinori Okada, Benjamin M Neale, and Mark J Daly. Clinical use of current polygenic risk scores may exacerbate health disparities. Nat. Genet. , 51(4):584-591, April 2019.
  63. Charles T Marx, Flavio du Pin Calmon, and Berk Ustun. Predictive multiplicity in classification. arXiv preprint arXiv:1909.06677 , 2019. · arXiv
  64. R Thomas McCoy, Junghyun Min, and Tal Linzen. Berts of a feather do not generalize together: Large variability in generalization across models with similar test set performance. arXiv preprint arXiv:1911.02969 , 2019a. · arXiv
  65. R Thomas McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv:1902.01007 , 2019b. · arXiv
  66. Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and double descent curve. arXiv:1908.05355 , 2019. · arXiv
  67. Joannella Morales, Danielle Welter, Emily H Bowler, Maria Cerezo, Laura W Harris, Aoife C McMahon, Peggy Hall, Heather A Junkins, Annalisa Milano, Emma Hastings, Cinzia Malangone, Annalisa Buniello, Tony Burdett, Paul Flicek, Helen Parkinson, Fiona Cunningham, Lucia A Hindorff, and Jacqueline A L MacArthur. A standardized framework for representation of ancestry data in genomics studies, with application to the NHGRI-EBI GWAS catalog. Genome Biol. , 19 (1):21, February 2018.
  68. Sendhil Mullainathan and Jann Spiess. Machine learning: an applied econometric approach. Journal of Economic Perspectives , 31(2):87-106, 2017.
  69. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. Stress test evaluation for natural language inference. arXiv preprint arXiv:1806.00692 , 2018. · arXiv
  70. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=B1g5sA4twr .
  71. Radford M Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks , pages 29-53. Springer, 1996.
  72. Anna C Need and David B Goldstein. Next generation disparities in human genomics: concerns and remedies. Trends Genet. , 25(11):489-494, November 2009.
  73. Bret Nestor, Matthew B. A. McDermott, Willie Boag, Gabriela Berner, Tristan Naumann, Michael C Hughes, Anna Goldenberg, and Marzyeh Ghassemi. Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks. Proceedings of Machine Learning Research , 106:1-23, 2019. URL https://mimic.physionet.org/ mimicdata/carevue/http://arxiv.org/abs/1908.00690 .
  74. Cecilia Panigutti, Alan Perotti, and Dino Pedreschi. Doctor XAI An ontology-based approach to blackbox sequential data classification explanations. In FAT* 2020 - Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , pages 629-639, 2020. ISBN 9781450369367. doi: 10.1145/3351095.3372855. URL https://doi.org/10.1145/3351095.3372855 . · DOI
  75. Jonas Peters, Peter Bühlmann, and Nicolai Meinshausen. Causal inference by using invariant prediction: identification and confidence intervals. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 78(5):947-1012, 2016. doi: 10.1111/rssb.12167. URL https://rss. onlinelibrary.wiley.com/doi/abs/10.1111/rssb.12167 . · DOI
  76. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 2227-2237, 2018.
  77. Alice B Popejoy and Stephanie M Fullerton. Genomics is failing on diversity. Nature , 538(7624): 161-164, October 2016.
  78. Mihail Popescu and Mohammad Khalilia. Improving disease prediction using ICD-9 ontological features. In IEEE International Conference on Fuzzy Systems , pages 1805-1809, 2011. ISBN 9781424473175. doi: 10.1109/FUZZY.2011.6007410. · DOI
  79. Alkes L Price, Nick J Patterson, Robert M Plenge, Michael E Weinblatt, Nancy A Shadick, and David Reich. Principal components analysis corrects for stratification in genome-wide association studies. Nat. Genet. , 38(8):904-909, August 2006.
  80. Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and accuracy. arXiv preprint arXiv:2002.10716 , 2020. · arXiv
  81. Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in neural information processing systems , pages 1177-1184, 2008.
  82. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) , pages 8-14, 2018.
  83. Bernhard Schölkopf. Causality for machine learning. arXiv preprint arXiv:1911.10500 , 2019. · arXiv
  84. Lesia Semenova, Cynthia Rudin, and Ronald Parr. A study in rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning. arXiv preprint arXiv:1908.01755 , 2019. · arXiv
  85. Cathie Sudlow, John Gallacher, Naomi Allen, Valerie Beral, Paul Burton, John Danesh, Paul Downey, Paul Elliott, Jane Green, Martin Landray, Bette Liu, Paul Matthews, Giok Ong, Jill Pell, Alan Silman, Alan Young, Tim Sprosen, Tim Peakman, and Rory Collins. UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age. PLoS Med. , 12(3):e1001779, March 2015.
  86. Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision , pages 843-852, 2017.
  87. Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inceptionresnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence , 2017.
  88. Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. arXiv preprint arXiv:2007.00644 , 2020. · arXiv
  89. Daniel Shu Wei Ting, Carol Yim-Lui Cheung, Gilbert Lim, Gavin Siew Wei Tan, Nguyen D Quang, Alfred Gan, Haslina Hamzah, Renata Garcia-Franco, Ian Yew San Yeo, Shu Yen Lee, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. Jama , 318(22):2211-2223, 2017.
  90. Nenad Tomašev, Xavier Glorot, Jack W Rae, Michal Zielinski, Harry Askham, Andre Saraiva, Anne Mottram, Clemens Meyer, Suman Ravuri, Ivan Protsyuk, Alistair Connell, Cían O Hughes, Alan Karthikesalingam, Julien Cornebise, Hugh Montgomery, Geraint Rees, Chris Laing, Clifton R Baker, Kelly Peterson, Ruth Reeves, Demis Hassabis, Dominic King, Mustafa Suleyman, Trevor Back, Christopher Nielson, Joseph R Ledsam, and Shakir Mohamed. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature , 572(7767):116-119, aug 2019a. ISSN 0028-0836. doi: 10.1038/s41586-019-1390-1. · DOI · Paper page
  91. Nenad Tomašev, Xavier Glorot, Jack W. Rae, Michal Zielinski, Harry Askham, Andre Saraiva, Anne Mottram, Clemens Meyer, Suman Ravuri, Ivan Protsyuk, Alistair Connell, Cian O. Hugues, Alan Kathikesalingam, Julien Cornebise, Hugh Montgomery, Geraint Rees, Chris Laing, Clifton R. Baker, Kelly Peterson, Ruth Reeves, Demis Hassabis, Dominic King, Mustafa Suleyman, Trevor Back, Christopher Nielson, Joseph R. Ledsam, and Shakir Mohamed. Developing Deep Learning Continuous Risk Models for Early Adverse Event Prediction in Electronic Health Records: an AKI Case Study. PROTOCOL available at Protocol Exchange , version 1, jul 2019b. doi: 10.21203/RS. 2.10083/V1. · DOI
  92. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998-6008, 2017.
  93. Bjarni J Vilhjálmsson, Jian Yang, Hilary K Finucane, Alexander Gusev, Sara Lindström, Stephan Ripke, Giulio Genovese, Po-Ru Loh, Gaurav Bhatia, Ron Do, Tristan Hayeck, Hong-Hee Won, Schizophrenia Working Group of the Psychiatric Genomics Consortium, Discovery, Biology, and Risk of Inherited Variants in Breast Cancer (DRIVE) study, Sekar Kathiresan, Michele Pato, Carlos Pato, Rulla Tamimi, Eli Stahl, Noah Zaitlen, Bogdan Pasaniuc, Gillian Belbin, Eimear E Kenny, Mikkel H Schierup, Philip De Jager, Nikolaos A Patsopoulos, Steve McCarroll, Mark Daly, Shaun Purcell, Daniel Chasman, Benjamin Neale, Michael Goddard, Peter M Visscher, Peter Kraft, Nick Patterson, and Alkes L Price. Modeling linkage disequilibrium increases accuracy of polygenic risk scores. Am. J. Hum. Genet. , 97(4):576-592, October 2015.
  94. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 353-355, 2018.
  95. Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems , pages 10506-10518, 2019.
  96. Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8684-8694, 2020.
  97. Kellie Webster, Xuezhi Wang, Ian Tenney, Alex Beutel, Emily Pitler, Ellie Pavlick, Jilin Chen, and Slav Petrov. Measuring and reducing gendered correlations in pre-trained models. arXiv preprint arXiv:2010.06032 , 2020. · arXiv
  98. Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. arXiv preprint arXiv:2006.13570 , 2020. · arXiv
  99. Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , pages 1112-1122, 2018.
  100. Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. arXiv preprint arXiv:2002.08791 , 2020. · arXiv
  101. Naomi R Wray, Michael E Goddard, and Peter M Visscher. Prediction of individual genetic risk to disease from genome-wide association studies. Genome Res. , 17(10):1520-1528, October 2007.
  102. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems , pages 5753-5763, 2019.
  103. Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. In Advances in Neural Information Processing Systems , pages 13255-13265, 2019.
  104. Bin Yu et al. Stability. Bernoulli , 19(4):1484-1500, 2013.
  105. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326 , 2018. · arXiv
  106. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) , pages 15-20, 2018.
  107. Xiang Zhou, Yixin Nie, Hao Tan, and Mohit Bansal. The curse of performance instability in analysis datasets: Consequences, source, and suggestions. arXiv preprint arXiv:2004.13606 , 2020. · arXiv

Cited by

Sorted by citations

Authors
D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., Hormozdiari, F., Houlsby, N., Hou, S., Jerfel, G., Karthikesalingam, A., Lucic, M., Ma, Y., McLean, C., Mincu, D., Mitani, A., Montanari, A., Nado, Z., Natarajan, V., Nielson, C., Osborne, T. F., Raman, R., Ramasamy, K., Sayres, R., Schrouff, J., Seneviratne, M., Sequeira, S., Suresh, H., Veitch, V., Vladymyrov, M., Wang, X., Webster, K., Yadlowsky, S., Yun, T., Zhai, X., & Sculley, D.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2011.03395
arXiv
2011.03395
OpenAlex
W3100511085
Cited by
398
References
27
Citations by year
2026: 3 · 2025: 42 · 2024: 66 · 2023: 84 · 2022: 78 · 2021: 111 · 2020: 13 · 2019: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
DeepMind
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
underspecification, model credibility, model instability, model selection, deep learning, model failure modes, clinical risk prediction, electronic health records, genomic medicine, computer vision, natural language processing, diagnostic imaging

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review