Conditional variance penalties and domain shift robustness
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
When training a deep neural network for image classification, one can broadly distinguish between two types of latent features of images that will drive the classification. We can divide latent features into (i) ‘core’ or ‘conditionally invariant’ features$$C$$Cwhose distribution$$C\vert Y$$C|Y, conditional on the class Y, does not change substantially across domains and (ii) ‘style’ features$$S$$Swhose distribution$$S\vert Y$$S|Ycan change substantially across domains. Examples for style features include position, rotation, image quality or brightness but also more complex ones like hair color, image quality or posture for images of persons. Our goal is to minimize a loss that is robust under changes in the distribution of these style features. In contrast to previous work, we assume that the domain itself is not observed and hence a latent variable. We do assume that we can sometimes observe a typically discrete identifier or “$$\mathrm {ID}$$IDvariable”. In some applications we know, for example, that two images show the same person, and$$\mathrm {ID}$$IDthen refers to the identity of the person. The proposed method requires only a small fraction of images to have$$\mathrm {ID}$$IDinformation. We group observations if they share the same class and identifier$$(Y,\mathrm {ID})=(y,\mathrm {id})$$(Y,ID)=(y,id)and penalize the conditional variance of the prediction or the loss if we condition on$$(Y,\mathrm {ID})$$(Y,ID). Using a causal framework, this conditional variance regularization (C o R e) is shown to protect asymptotically against shifts in the distribution of the style variables in a partially linear structural equation model. Empirically, we show that the C o R e penalty improves predictive accuracy substantially in settings where domain changes occur in terms of image quality, brightness and color while we also look at more complex changes such as changes in movement and posture.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
Linear Mixed Models for Longitudinal Data
Verbeke, G. (1997) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-2294-1_3 · Cited by 3,506
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 2×
“As any bias in the training data is in general used to discriminate between classes, these biases will persist in future classifications, raising also considerations of fairness and discrimination (Barocas and Selbst 2016).”
BackgroundTorralba, A., & Efros, A. A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995347 · Cited by 2,440 · cited 1×
“Issues can arise when the learned representations rely on dependencies that vanish in test distributions, see for example Quionero-Candela et al. (2009); Torralba and Efros (2011); Csurka (2017) and references therein.”
Background
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
Toward Causal Representation Learning
Scholkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., & Bengio, Y. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3058954 · Cited by 1,170
Generalizing to unseen domains: A survey on domain generalization
Wang, J., Lan, C., Liu, C., Ouyang, Y., Qin, T., Lu, W., Chen, Y., Zeng, W., & Yu, P. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3178128 · Cited by 1,046
“Using the additional data of object identity (it is a more detailed label than the class label), [206] enforced the conditional”
BackgroundArjovsky, M., Bottou, L., Gulrajani, I., & Lopez-Paz, D. (2019) · arXiv · doi:10.48550/arxiv.1907.02893 · Cited by 927
“Some works in machine learning [45, 18, 21, 26, 36, 43, 34, 7] pursue similar questions.”
BackgroundOut-of-Distribution Generalization via Risk Extrapolation (REx)
Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Binas, J., Zhang, D., Priol, R. L., & Courville, A. (2020) · arXiv · doi:10.48550/arxiv.2003.00688 · Cited by 407
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518
Causality Inspired Representation Learning for Domain Generalization
Lv, F., Liang, J., Li, S., Zang, B., Liu, C. H., Wang, Z., & Liu, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00788 · Cited by 224
Iyer, J. S., Juyal, D., Le, Q., Shanis, Z., Pokkalla, H., Pouryahya, M., Pedawi, A., Stanford-Moore, S. A., Biddle-Snead, C., Carrasco-Zevallos, O., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-03172-7 · Cited by 54
Metadata
- Authors
- Heinze-Deml, C., & Meinshausen, N.
- Venue
- Machine Learning
- Published
- 2020-11-23
- Volume
- 110
- Issue
- 2
- Pages
- 303-348
- ISSN
- 0885-6125 (ISSN-L)
- 1573-0565
- OpenAlex
- W2795857097
- Cited by
- 97
- References
- 18
- FWCI
- 7.58
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.