CoChat
Sign Up
PreprintCited by 393Domain Adaptation and Few-Shot LearningData Stream Mining TechniquesMachine Learning and Data Classification

Test-Time Training with Self-Supervision for Generalization under Distribution Shifts

Open access
Sun, Yu, Wang, Xiaolong, Liu, Zhuang, Miller, John, Efros, Alexei A., Hardt, Moritz·2019·arXiv
DOI 10.48550/arxiv.1909.13231arXiv 1909.13231
Open this paper in CoChat — free accountOpen PDF

Abstract

In this paper, we propose Test-Time Training, a general approach for improving the performance of predictive models when training and test data come from different distributions. We turn a single unlabeled test sample into a self-supervised learning problem, on which we update the model parameters before making a prediction. This also extends naturally to data in an online stream. Our simple approach leads to improvements on diverse image classification benchmarks aimed at evaluating robustness to distribution shifts.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1909.13231

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Test-time training makes a subset of the model's parameters trainable at inference time, so that the weights themselves act as contextual memory.

    “Test-time training (TTT) [42, 43] offers a complementary perspective by making a subset of the model's parameters trainable at inference time, so that the weights themselves act as contextual memory.”

    From Learning What to Remember: Test-Time Training via Context Distillation (2026) · cites this paper as [42]

    The Method (p. 2) says test-time training updates the shared feature extractor’s parameters before prediction, and its online version carries updated parameters forward so they can use information from earlier samples. But the standard version discards those updates after each prediction, so describing the weights as persistent contextual memory does not apply to TTT generally. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 91 · Cited by 393

Research works printed but not linked: 82

  1. Asano, Y. M., Rupprecht, C., and Vedaldi, A. Surprising effectiveness of few-image unsupervised feature learning. arXiv preprint arXiv:1904.13132 , 2019. · arXiv
  2. Athalye, A., Carlini, N., and Wagner, D. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.00420 , 2018. · arXiv
  3. Balaji, Y ., Sankaranarayanan, S., and Chellappa, R. Metareg: Towards domain generalization using meta-regularization. In Advances in Neural Information Processing Systems , pp. 998-1008, 2018.
  4. Bau, D., Strobelt, H., Peebles, W., Wulff, J., Zhou, B., Zhu, J.-Y., and Torralba, A. Semantic photo manipulation with a generative image prior. ACM Transactions on Graphics (TOG) , 38(4):59, 2019.
  5. Bojanowski, P. and Joulin, A. Unsupervised learning by predicting noise. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pp. 517526. JMLR. org, 2017.
  6. Carlucci, F. M., D'Innocente, A., Bucci, S., Caputo, B., and Tommasi, T. Domain generalization by solving jigsaw puzzles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 2229-2238, 2019.
  7. Caron, M., Bojanowski, P., Joulin, A., and Douze, M. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV) , pp. 132-149, 2018.
  8. Chen, M., Weinberger, K. Q., and Blitzer, J. Co-training for domain adaptation. In Advances in neural information processing systems , pp. 2456-2464, 2011.
  9. Chen, X., Sun, Y., Athiwaratkun, B., Cardie, C., and Weinberger, K. Adversarial deep averaging networks for crosslingual sentiment classification. Transactions of the Association for Computational Linguistics , 6:557-570, 2018.
  10. Cohen, J. M., Rosenfeld, E., and Kolter, J. Z. Certified adversarial robustness via randomized smoothing. arXiv preprint arXiv:1902.02918 , 2019. · arXiv
  11. Croce, F., Andriushchenko, M., and Hein, M. Provable robustness of relu networks via maximization of linear regions. arXiv preprint arXiv:1810.07481 , 2018. · arXiv
  12. Csurka, G. Domain adaptation for visual applications: A comprehensive survey. arXiv preprint arXiv:1702.05374 , 2017. · arXiv
  13. Ding, G. W., Wang, L., and Jin, X. AdverTorch v0.1: An adversarial robustness toolbox based on pytorch. arXiv preprint arXiv:1902.07623 , 2019. · arXiv
  14. Doersch, C., Gupta, A., and Efros, A. A. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision , pp. 1422-1430, 2015.
  15. Fei-Fei, L., Fergus, R., and Perona, P. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence , 28(4):594-611, 2006.
  16. Finn, C., Abbeel, P., and Levine, S. Model-agnostic metalearning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pp. 1126-1135. JMLR. org, 2017.
  17. Gammerman, A., Vovk, V., and Vapnik, V. Learning by transduction. In Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence , pp. 148-155. Morgan Kaufmann Publishers Inc., 1998.
  18. Gan, C., Yang, T., and Gong, B. Learning attributes equals multi-source domain generalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 87-97, 2016.
  19. Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. Domain-adversarial training of neural networks. The Journal of Machine Learning Research , 17(1):2096-2030, 2016.
  20. Geirhos, R., Temme, C. R., Rauber, J., Sch¨ utt, H. H., Bethge, M., and Wichmann, F. A. Generalisation in humans and deep neural networks. In Advances in Neural Information Processing Systems , pp. 7538-7550, 2018.
  21. Ghifary, M., Bastiaan Kleijn, W., Zhang, M., and Balduzzi, D. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision , pp. 25512559, 2015.
  22. Gidaris, S. and Komodakis, N. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 4367-4375, 2018.
  23. Gidaris, S., Singh, P., and Komodakis, N. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728 , 2018. · arXiv
  24. Gong, B., Shi, Y., Sha, F., and Grauman, K. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition , pp. 2066-2073. IEEE, 2012.
  25. Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014. · arXiv
  26. Guo, C., Rana, M., Cisse, M., and van der Maaten, L. Countering adversarial images using input transformations. arXiv preprint arXiv:1711.00117 , 2017. · arXiv
  27. Hazan, E. et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157325, 2016.
  28. He, K., Zhang, X., Ren, S., and Sun, J. Identity mappings in deep residual networks. In European conference on computer vision , pp. 630-645. Springer, 2016b.
  29. He, K., Girshick, R., and Doll´ ar, P. Rethinking imagenet pre-training. arXiv preprint arXiv:1811.08883 , 2018. · arXiv
  30. Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 , 2019. · arXiv
  31. Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136 , 2016. · arXiv
  32. Hendrycks, D., Mazeika, M., Wilson, D., and Gimpel, K. Using trusted data to train deep networks on labels corrupted by severe noise. In Advances in neural information processing systems , pp. 10456-10465, 2018.
  33. Hendrycks, D., Lee, K., and Mazeika, M. Using pre-training can improve model robustness and uncertainty. arXiv preprint arXiv:1901.09960 , 2019a. · arXiv
  34. Hendrycks, D., Mazeika, M., Kadavath, S., and Song, D. Improving model robustness and uncertainty estimates with self-supervised learning. arXiv preprint , 2019b.
  35. Hoffman, J., Kulis, B., Darrell, T., and Saenko, K. Discovering latent domains for multisource domain adaptation. In European Conference on Computer Vision , pp. 702-715. Springer, 2012.
  36. Hoffman, J., Darrell, T., and Saenko, K. Continuous manifold based adaptation for evolving visual domains. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 867-874, 2014.
  37. Hoffman, J., Tzeng, E., Park, T., Zhu, J.-Y., Isola, P., Saenko, K., Efros, A. A., and Darrell, T. Cycada: Cycleconsistent adversarial domain adaptation. arXiv preprint arXiv:1711.03213 , 2017. · arXiv
  38. Hoffman, J., Mohri, M., and Zhang, N. Algorithms and theory for multiple-source adaptation. In Advances in Neural Information Processing Systems , pp. 8246-8256, 2018.
  39. Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger, K. Q. Deep networks with stochastic depth. In European conference on computer vision , pp. 646-661. Springer, 2016.
  40. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 , 2015. · arXiv
  41. Jain, V. and Learned-Miller, E. Online domain adaptation of a pre-trained cascade of classifiers. In CVPR 2011 , pp. 577-584. IEEE, 2011.
  42. Kalal, Z., Mikolajczyk, K., and Matas, J. Tracking-learningdetection. IEEE transactions on pattern analysis and machine intelligence , 34(7):1409-1422, 2011.
  43. Kang, D., Sun, Y., Brown, T., Hendrycks, D., and Steinhardt, J. Transfer of adversarial robustness between perturbation types. arXiv preprint arXiv:1905.01034 , 2019. · arXiv
  44. Kannan, H., Kurakin, A., and Goodfellow, I. Adversarial logit pairing. arXiv preprint arXiv:1803.06373 , 2018. · arXiv
  45. Larsson, G., Maire, M., and Shakhnarovich, G. Colorization as a proxy task for visual understanding. In CVPR , 2017.
  46. Li, D., Yang, Y., Song, Y .-Z., and Hospedales, T. M. Deeper, broader and artier domain generalization. In Proceedings of the IEEE International Conference on Computer Vision , pp. 5542-5550, 2017a.
  47. Li, D., Zhang, J., Yang, Y., Liu, C., Song, Y.-Z., and Hospedales, T. M. Episodic training for domain generalization. arXiv preprint arXiv:1902.00113 , 2019. · arXiv
  48. Li, Y., Tian, X., Gong, M., Liu, Y., Liu, T., Zhang, K., and Tao, D. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV) , pp. 624-639, 2018.
  49. Li, Z. and Hoiem, D. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence , 40(12):2935-2947, 2017.
  50. Li, Z., Zhou, F., Chen, F., and Li, H. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09835 , 2017b. · arXiv
  51. Liu, Z., Sun, M., Zhou, T., Huang, G., and Darrell, T. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270 , 2018. · arXiv
  52. Long, M., Cao, Y., Wang, J., and Jordan, M. I. Learning transferable features with deep adaptation networks. arXiv preprint arXiv:1502.02791 , 2015. · arXiv
  53. Long, M., Zhu, H., Wang, J., and Jordan, M. I. Unsupervised domain adaptation with residual transfer networks. In Advances in Neural Information Processing Systems , pp. 136-144, 2016.
  54. Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems , pp. 6467-6476, 2017.
  55. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017. · arXiv
  56. Motiian, S., Piccirilli, M., Adjeroh, D. A., and Doretto, G. Unified deep supervised domain adaptation and generalization. In Proceedings of the IEEE International Conference on Computer Vision , pp. 5715-5725, 2017.
  57. Mullapudi, R. T., Chen, S., Zhang, K., Ramanan, D., and Fatahalian, K. Online model distillation for efficient video inference. arXiv preprint arXiv:1812.02699 , 2018. · arXiv
  58. Noroozi, M. and Favaro, P. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision , pp. 69-84. Springer, 2016.
  59. Raghunathan, A., Steinhardt, J., and Liang, P. Certified defenses against adversarial examples. arXiv preprint arXiv:1801.09344 , 2018. · arXiv
  60. Ravi, S. and Larochelle, H. Optimization as a model for few-shot learning. IEEE transactions on pattern analysis and machine intelligence , 2016.
  61. Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv:1806.00451 , 2018. · arXiv
  62. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) , 115(3):211-252, 2015. doi: 10.1007/s11263-015-0816-y. · DOI · Paper page
  63. Salman, H., Yang, G., Li, J., Zhang, P., Zhang, H., Razenshteyn, I., and Bubeck, S. Provably robust deep learning via adversarially trained smoothed classifiers. arXiv preprint arXiv:1906.04584 , 2019. · arXiv
  64. Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., and Lillicrap, T. Meta-learning with memory-augmented neural networks. In International conference on machine learning , pp. 1842-1850, 2016.
  65. Shalev-Shwartz, S. et al. Online learning and online convex optimization. Foundations and Trends® in Machine Learning , 4(2):107-194, 2012.
  66. Shankar, S., Piratla, V., Chakrabarti, S., Chaudhuri, S., Jyothi, P., and Sarawagi, S. Generalizing across domains via cross-gradient training. arXiv preprint arXiv:1804.10745 , 2018. · arXiv
  67. Shankar, V., Dave, A., Roelofs, R., Ramanan, D., Recht, B., and Schmidt, L. Do image classifiers generalize across time? arXiv , 2019.
  68. Shocher, A., Cohen, N., and Irani, M. zero-shot superresolution using deep internal learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 3118-3126, 2018.
  69. Sinha, A., Namkoong, H., and Duchi, J. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv:1710.10571 , 2017. · arXiv
  70. Snell, J., Swersky, K., and Zemel, R. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems , pp. 4077-4087, 2017.
  71. Song, Y., Kim, T., Nowozin, S., Ermon, S., and Kushman, N. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. arXiv preprint arXiv:1710.10766 , 2017. · arXiv
  72. Su, J.-C., Maji, S., and Hariharan, B. Boosting supervision with self-supervision for few-shot learning. arXiv preprint arXiv:1906.07079 , 2019. · arXiv
  73. Sun, Y., Tzeng, E., Darrell, T., and Efros, A. A. Unsupervised domain adaptation through self-supervision. arXiv preprint , 2019.
  74. Tripuraneni, N. and Mackey, L. Debiasing linear prediction. arXiv preprint arXiv:1908.02341 , 2019. · arXiv
  75. Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., and Madry, A. Robustness may be at odds with accuracy. arXiv preprint arXiv:1805.12152 , 2018. · arXiv
  76. Vapnik, V. The nature of statistical learning theory . Springer science & business media, 2013.
  77. Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al. Matching networks for one shot learning. In Advances in neural information processing systems , pp. 3630-3638, 2016.
  78. Viola, P., Jones, M., et al. Rapid object detection using a boosted cascade of simple features. CVPR (1) , 1(511518):3, 2001.
  79. Wong, E. and Kolter, J. Z. Provable defenses against adversarial examples via the convex outer adversarial polytope. arXiv preprint arXiv:1711.00851 , 2017. · arXiv
  80. Zhang, H., Yu, Y., Jiao, J., Xing, E. P., Ghaoui, L. E., and Jordan, M. I. Theoretically principled trade-off between robustness and accuracy. arXiv preprint arXiv:1901.08573 , 2019. · arXiv
  81. Zhang, R., Isola, P., and Efros, A. A. Colorful image colorization. In European conference on computer vision , pp. 649-666. Springer, 2016.
  82. Zhu, W., Huang, Y., Vannan, M. A., Liu, S., Xu, D., Fan, W., Qian, Z., and Xie, X. Neural multi-scale self-supervised registration for echocardiogram dense tracking. arXiv preprint arXiv:1906.07357 , 2019. · arXiv

Authors
Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. A., & Hardt, M.
Affiliations
  • University of California, Berkeley (US)
  • Shanghai University (CN)
  • Intel (United States) (US)
  • Baidu (China) (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1909.13231
arXiv
1909.13231
OpenAlex
W3038628659
Cited by
393
References
91
Citations by year
2026: 1 · 2025: 71 · 2024: 104 · 2023: 94 · 2022: 47 · 2021: 60 · 2020: 15 · 2019: 1
Type
posted content
Language
en
Open access
Licence
creativecommons.org/publicdomain/zero/1.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (public-domain)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Data Stream Mining Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
Keywords
test-time training, self-supervised learning, distribution shift, domain generalization, online learning, image classification, distribution shift robustness, unlabeled test data

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review