CoChat
Sign Up
PreprintCited by 48Adversarial Robustness in Machine LearningDomain Adaptation and Few-Shot LearningMachine Learning and Data Classification

Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks

Open access
Malinin, Andrey, Band, Neil, Ganshin, Alexander, Chesnokov, German, Gal, Yarin, Gales, Mark J. F., Noskov, Alexey, Ploskonosov, Andrey +8 more·2021·arXiv
DOI 10.48550/arxiv.2107.07455arXiv 2107.07455
Open this paper in CoChat — free accountOpen PDF

Abstract

There has been significant research done on developing methods for improving robustness to distributional shift and uncertainty estimation. In contrast, only limited work has examined developing standard datasets and benchmarks for assessing these approaches. Additionally, most work on uncertainty estimation and robustness has developed new techniques based on small-scale regression or image classification tasks. However, many tasks of practical interest have different modalities, such as tabular data, audio, text, or sensor data, which offer significant challenges involving regression and discrete or continuous structured prediction. Thus, given the current state of the field, a standardized large-scale dataset of tasks across a range of modalities affected by distributional shifts is necessary. This will enable researchers to meaningfully evaluate the plethora of recently developed uncertainty quantification methods, as well as assessment criteria and state-of-the-art baselines. In this work, we propose the Shifts Dataset for evaluation of uncertainty estimates and robustness to distributional shift. The dataset, which has been collected from industrial sources and services, is composed of three tasks, with each corresponding to a particular data modality: tabular weather prediction, machine translation, and self-driving car (SDC) vehicle motion prediction. All of these data modalities and tasks are affected by real, "in-the-wild" distributional shifts and pose interesting challenges with respect to uncertainty estimation. In this work we provide a description of the dataset and baseline results for all tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2107.07455

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Only the Lyft, Shifts and nuPlan datasets provide more than 1000h of driving data, and nuPlan was the first such dataset that provides lidar and camera sensor data (128h), although Waymo later also released compressed lidar data.

    “Only the Lyft [19], Shifts [15] and nuPlan datasets provide more than 1000h of driving data and nuPlan was the first such dataset that provides lidar and camera sensor data (128h), although Waymo [17] later also released compressed lidar data.”

    From Towards learning-based planning: The nuPlan benchmark for real-world autonomous driving (2024) · cites this paper as [15]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 56

  1. [1][1] Karen Simonyan and Andrew Zisserman, 'Very Deep Convolutional Networks for Large-Scale Image Recognition,' in Proc. International Conference on Learning Representations (ICLR) , 2015. -
  2. [2][2] Tomas Mikolov et al., 'Linguistic Regularities in Continuous Space Word Representations,' in Proc. NAACL-HLT , 2013. -
  3. [3][3] Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjeev Khudanpur, 'Recurrent Neural Network Based Language Model,' in Proc. INTERSPEECH , 2010. -
  4. [4][4] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, 'Neural machine translation by jointly learning to align and translate,' in Proc. International Conference on Learning Representations (ICLR) , 2015. -
  5. [5][5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin, 'Attention is all you need,' in Advances in neural information processing systems , 2017, pp. 5998-6008. -
  6. [7][7] Joaquin Quiñonero-Candela, Dataset Shift in Machine Learning , The MIT Press, 2009. -
  7. [10][10] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané, 'Concrete problems in AI safety,' http://arxiv.org/abs/1606.06565 , 2016, arXiv: 1606.06565. - · arXiv
  8. [11][11] Yarin Gal, Uncertainty in Deep Learning , Ph.D. thesis, University of Cambridge, 2016. -
  9. [12][12] Andrey Malinin, Uncertainty Estimation in Deep Learning with application to Spoken Language Assessment , Ph.D. thesis, University of Cambridge, 2019. -
  10. [14][14] B. Lakshminarayanan, A. Pritzel, and C. Blundell, 'Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles,' in Proc. Conference on Neural Information Processing Systems (NIPS) , 2017. -
  11. [15][15] Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov, 'Pitfalls of indomain uncertainty estimation and ensembling in deep learning,' in International Conference on Learning Representations , 2020. -
  12. [17][17] Nicholas Carlini and David A. Wagner, 'Adversarial examples are not easily detected: Bypassing ten detection methods,' CoRR , 2017. -
  13. [18][18] L. Smith and Y. Gal, 'Understanding Measures of Uncertainty for Adversarial Example Detection,' in UAI , 2018. -
  14. [19][19] Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal, 'Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning,' Advances in neural information processing systems , vol. 32, pp. 7026-7037, 2019. -
  15. [20][20] Tim Z Xiao, Aidan N Gomez, and Yarin Gal, 'Wat heb je gezegd? detecting out-of-distribution translations with variational transformers,' in Bayesian Deep Learning Workshop (NeurIPS) , 2019. -
  16. [21][21] Pascal Notin, José Miguel Hernández-Lobato, and Yarin Gal, 'Principled uncertainty estimation for high dimensional data,' in Uncertainty & Robustness in Deep Learning Workshop, ICML , 2020. -
  17. [27][27] Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal, 'Uncertainty estimation using a single deep deterministic neural network,' in International Conference on Machine Learning . PMLR, 2020, pp. 9690-9700. -
  18. [29][29] Jeremiah Zhe Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax-Weiss, and Balaji Lakshminarayanan, 'Simple and principled uncertainty estimation with deterministic deep learning via distance awareness,' arXiv preprint arXiv:2006.10108 , 2020. - · arXiv
  19. [30][30] Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal, 'Improving deterministic uncertainty estimation in deep learning for classification and regression,' arXiv preprint arXiv:2102.11409 , 2021. - · arXiv
  20. [31][31] Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal, 'Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty,' arXiv preprint arXiv:2102.11582 , 2021. - · arXiv
  21. [32][32] Andrey Malinin and Mark Gales, 'Predictive uncertainty estimation via prior networks,' in Advances in Neural Information Processing Systems , 2018, pp. 7047-7058. -
  22. [35][35] Andrey Malinin, Bruno Mlodozeniec, and Mark JF Gales, 'Ensemble distribution distillation,' in International Conference on Learning Representations , 2020. -
  23. [36][36] Max Ryabinin, Andrey Malinin, and Mark Gales, 'Scaling ensemble distribution distillation to many classes with proxy targets,' arXiv preprint arXiv:2105.06987 , 2021. - · arXiv
  24. [38][38] Neil Band, Tim G. J. Rudner, Qixuan Feng, Angelos Filos, Zachary Nado, Michael W. Dusenberry, Ghassen Jerfel, Dustin Tran, and Yarin Gal, 'Benchmarking bayesian deep learning on diabetic retinopathy detection tasks,' in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2021. -
  25. [43][43] Paul Michel and Graham Neubig, 'MTNT: A testbed for Machine Translation of Noisy Text,' in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2018. -
  26. [44][44] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, 'Gradient-based learning applied to document recognition,' Proceedings of the IEEE, vol. 86, pp. 2278-2324, 1998. -
  27. [48][48] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz, 'Invariant risk minimization,' arXiv preprint arXiv:1907.02893 , 2019. - · arXiv
  28. [49][49] João Gama, Indr˙ e Žliobait˙ e, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia, 'A survey on concept drift adaptation,' ACM computing surveys (CSUR) , vol. 46, no. 4, pp. 1-37, 2014. -
  29. [50][50] Alexey Tsymbal, 'The problem of concept drift: definitions and related work,' Computer Science Department, Trinity College Dublin , vol. 106, no. 2, pp. 58, 2004. -
  30. [51][51] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin, 'Catboost: unbiased boosting with categorical features,' in Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS) , 2018, pp. 6638-6648. -
  31. [52][52] Andrey Malinin, Liudmila Prokhorenkova, and Aleksei Ustimenko, 'Uncertainty in gradient boosting via ensembles,' in International Conference on Learning Representations , 2021. -
  32. [53][53] Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun, 'Improving backtranslation with uncertainty-based confidence estimation,' arXiv preprint arXiv:1909.00157 , 2019. - · arXiv
  33. [55][55] Matt Post, 'A call for clarity in reporting BLEU scores,' in Proceedings of the Third Conference on Machine Translation: Research Papers , Belgium, Brussels, Oct. 2018, pp. 186-191, Association for Computational Linguistics. -
  34. [56][56] Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault, 'Ground truth for grammatical error correction metrics,' in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on - Natural Language Processing (Volume 2: Short Papers) , Beijing, China, July 2015, pp. 588593, Association for Computational Linguistics. -
  35. [57][57] Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault, 'GLEU without tuning,' eprint arXiv:1605.02592 [cs.CL] , 2016. - · arXiv
  36. [59][59] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli, 'fairseq: A fast, extensible toolkit for sequence modeling,' in Proceedings of NAACL-HLT 2019: Demonstrations , 2019. -
  37. [60][60] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese, 'Social lstm: Human trajectory prediction in crowded spaces,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 961-971. -
  38. [61][61] Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker, 'Desire: Distant future prediction in dynamic scenes with interacting agents,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017, pp. 336-345. -
  39. [62][62] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi, 'Social gan: Socially acceptable trajectories with generative adversarial networks,' in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2018, pp. 2255-2264. -
  40. [63][63] Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov, 'Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction,' arXiv preprint arXiv:1910.05449 , 2019. - · arXiv
  41. [64][64] Sergio Casas, Cole Gulino, Renjie Liao, and Raquel Urtasun, 'Spatially-aware graph neural networks for relational behavior forecasting from sensor data,' arXiv preprint arXiv:1910.08233 , 2019. - · arXiv
  42. [65][65] Henggang Cui, Vladan Radosavljevic, Fang-Chieh Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schneider, and Nemanja Djuric, 'Multimodal trajectory predictions for autonomous driving using deep convolutional networks,' in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019, pp. 2090-2096. -
  43. [66][66] Tung Phan-Minh, Elena Corina Grigore, Freddy A Boulton, Oscar Beijbom, and Eric M Wolff, 'Covernet: Multimodal behavior prediction using trajectory sets,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 14074-14083. -
  44. [67][67] Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid, 'Vectornet: Encoding hd maps and agent dynamics from vectorized representation,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2020, pp. 11525-11533. -
  45. [68][68] Yuriy Biktairov, Maxim Stebelev, Irina Rudenko, Oleh Shliazhko, and Boris Yangel, 'Prank: motion prediction based on ranking,' in Advances in Neural Information Processing Systems , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds. 2020, vol. 33, pp. 2553-2563, Curran Associates, Inc. -
  46. [69][69] Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie Liao, and Raquel Urtasun, 'Implicit latent variable model for scene-consistent motion forecasting,' in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16 . Springer, 2020, pp. 624-641. -
  47. [70][70] Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun, 'Learning lane graph representations for motion forecasting,' in European Conference on Computer Vision . Springer, 2020, pp. 541-556. -
  48. [71][71] Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, and Bolei Zhou, 'Multimodal motion prediction with stacked transformers,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, pp. 7577-7586. -
  49. [72][72] Angelos Filos, Panagiotis Tigkas, Rowan McAllister, Nicholas Rhinehart, Sergey Levine, and Yarin Gal, 'Can autonomous vehicles identify, recover from, and adapt to distribution shifts?,' in International Conference on Machine Learning . PMLR, 2020, pp. 3145-3153. -
  50. [73][73] Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy, 'End-to-end driving via conditional imitation learning,' in 2018 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2018, pp. 4693-4700. -
  51. [74][74] Nicholas Rhinehart, Rowan McAllister, and Sergey Levine, 'Deep imitative models for flexible inference, planning, and control,' CoRR , vol. abs/1810.06544, 2018. - · arXiv
  52. [76][76] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford, 'Datasheets for datasets,' arXiv preprint arXiv:1803.09010 , 2018. - · arXiv
  53. [77][77] Deliang Chen and Hans Weiteng Chen, 'Using the köppen classification to quantify climate variation and change: An example for 1901-2010,' Environmental Development , vol. 6, pp. 69-79, 2013. -
  54. [78][78] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko, 'Revisiting deep learning models for tabular data,' arXiv preprint arXiv:2106.11959 , 2021. - · arXiv
  55. [80][80] Abraham Wald, 'Contributions to the Theory of Statistical Estimation and Testing Hypotheses,' The Annals of Mathematical Statistics , vol. 10, no. 4, pp. 299 - 326, 1939. -
  56. [81][81] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio, 'Learning phrase representations using rnn encoderdecoder for statistical machine translation,' arXiv preprint arXiv:1406.1078 , 2014. - · arXiv

Cited by

Sorted by citations

Authors
Malinin, A., Band, N., Ganshin, A., Chesnokov, G., Gal, Y., Gales, M. J. F., Noskov, A., Ploskonosov, A., Prokhorenkova, L., Provilkov, I., Raina, V., Raina, V., Roginskiy, D., Shmatova, M., Tigas, P., & Yangel, B.
Venue
arXiv
Publisher
arXiv
Published
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2107.07455
arXiv
2107.07455
OpenAlex
W3180750393
Cited by
48
References
11
Citations by year
2026: 1 · 2025: 4 · 2024: 12 · 2023: 21 · 2022: 9 · 2021: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • neural information processing systems
Funders
University of Cambridge
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
distribution shift, uncertainty estimation, benchmark dataset, robustness, machine translation, structured prediction, in-the-wild data, industrial data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review