Invariant Risk Minimization
Open accessAbstract
We introduce Invariant Risk Minimization (IRM), a learning paradigm to estimate invariant correlations across multiple training distributions. To achieve this goal, IRM learns a data representation such that the optimal classifier, on top of that data representation, matches for all training distributions. Through theory and experiments, we show how the invariances learned by IRM relate to the causal structures governing the data and enable out-of-distribution generalization.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Many recent papers have studied datasets with shifts induced by synthetic transformations, such as changing the color of MNIST digits, or by disparate data splits, such as generalizing from cartoons to photos.
“many recent papers have studied datasets with shifts induced by synthetic transformations, such as changing the color of MNIST digits (Arjovsky et al., 2019), or by disparate data splits, such as generalizing from cartoons to photos (Li et al., 2017a).”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Arjovsky et al., 2019
Prior work on retrofitting widely used datasets with distribution shifts has focused on shifts that are cleanly characterized but not always likely to arise in real-world deployments.
“prior work on retrofitting them with distribution shifts has focused on shifts that are cleanly characterized but not always likely to arise in real-world deployments.”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Arjovsky et al., 2019
IRM penalizes feature distributions that have different optimal linear classifiers for each domain.
“IRM (Arjovsky et al., 2019), which penalizes feature distributions that have different optimal linear classifiers for each domain.”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Arjovsky et al., 2019
Colored MNIST is a variant of the MNIST handwritten digit classification dataset.
“Colored MNIST [Arjovsky et al., 2019] is a variant of the MNIST handwritten digit classification dataset [LeCun, 1998].”
From In Search of Lost Domain Generalization (2020) · cites this paper as [Arjovsky et al., 2019]
Colored MNIST examples have dimension (2,28,28).
“This dataset contains 70 , 000 examples of dimension (2 , 28 , 28) and 2 classes.”
From In Search of Lost Domain Generalization (2020) · cites this paper as [Arjovsky et al., 2019]
Colored MNIST has 2 classes.
“This dataset contains 70 , 000 examples of dimension (2 , 28 , 28) and 2 classes.”
From In Search of Lost Domain Generalization (2020) · cites this paper as [Arjovsky et al., 2019]
In Colored MNIST, the label is a noisy function of the digit and color, such that color bears correlation d with the label.
“The label is a noisy function of the digit and color, such that color bears correlation d with the label and the digit bears correlation 0.75 with the label.”
From In Search of Lost Domain Generalization (2020) · cites this paper as [Arjovsky et al., 2019]
Colored MNIST contains 70,000 examples.
“This dataset contains 70 , 000 examples of dimension (2 , 28 , 28) and 2 classes.”
From In Search of Lost Domain Generalization (2020) · cites this paper as [Arjovsky et al., 2019]
In Colored MNIST, the digit bears correlation 0.75 with the label.
“The label is a noisy function of the digit and color, such that color bears correlation d with the label and the digit bears correlation 0.75 with the label.”
From In Search of Lost Domain Generalization (2020) · cites this paper as [Arjovsky et al., 2019]
In Colored MNIST, domain d ∈ {0.1, 0.3, 0.9} contains a disjoint set of digits colored either red or blue.
“Domain d ∈ { 0 . 1 , 0 . 3 , 0 . 9 } contains a disjoint set of digits colored either red or blue.”
From In Search of Lost Domain Generalization (2020) · cites this paper as [Arjovsky et al., 2019]
Cites
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653 · see citing works
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · see citing works
Torralba, A., & Efros, A. A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995347 · Cited by 2,440 · see citing works
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., & Brendel, W. (2018) · arXiv · doi:10.48550/arxiv.1811.12231 · Cited by 1,490 · cited 1× · see citing works
“Unfortunately, spurious correlations and biases are often simpler to detect than the true phenomenon of interest [17, 9, 10, 11].”
BackgroundDeep Learning: A Critical Appraisal
Marcus, G. (2018) · arXiv · doi:10.48550/arxiv.1801.00631 · Cited by 853 · see citing works
Support and Invertibility in Domain-Invariant Representations
Johansson, F. D., Sontag, D., & Ranganath, R. (2019) · arXiv · doi:10.48550/arxiv.1903.03448 · Cited by 70 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224 · see citing works
Generalizing to unseen domains: A survey on domain generalization
Wang, J., Lan, C., Liu, C., Ouyang, Y., Qin, T., Lu, W., Chen, Y., Zeng, W., & Yu, P. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3178128 · Cited by 1,046 · see citing works
Mapping single-cell data to reference atlases by transfer learning
Lotfollahi, M., Naghipourfar, M., Luecken, M. D., Khajavi, M., Büttner, M., Wagenstetter, M., Avsec, Ž., Gayoso, A., Yosef, N., Interlandi, M., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01001-7 · Cited by 602 · see citing works
Causality matters in medical imaging
Castro, D. C., Walker, I., & Glocker, B. (2020) · Nature Communications · doi:10.1038/s41467-020-17478-w · Cited by 364 · see citing works
WILDS: A Benchmark of in-the-Wild Distribution Shifts
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L., Gao, I., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07421 · Cited by 518 · see citing works
“Numerous methods have been proposed for domain generalization, including domain-invariant learning (Ganin et al., 2016; Sun and Saenko, 2016), invariant risk minimization (IRM) (Arjovsky et al., 2019), and meta-learning-based methods (Li et al., 2017; Dou et al., 2019).”
InfluentialA Survey of Learning Causality with Data: Problems and Methods
Guo, R., Cheng, L., Li, J., Hahn, P. R., & Liu, H. (2021) · ACM Computing Surveys · doi:10.1145/3397269 · Cited by 362 · see citing works
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond
Feder, A., Keith, K. A., Manzoor, E., Pryzant, R., Sridhar, D., Wood-Doughty, Z., Eisenstein, J., Grimmer, J., Reichart, R., Roberts, M. E., et al. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00511 · Cited by 220 · see citing works
A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
Kirk, R., Zhang, A., Grefenstette, E., & Rocktäschel, T. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14174 · Cited by 193 · see citing works
Guo, L. L., Pfohl, S. R., Fries, J., Johnson, A. E. W., Posada, J., Aftandilian, C., Shah, N., & Sung, L. (2022) · Scientific Reports · doi:10.1038/s41598-022-06484-1 · Cited by 103 · see citing works
“LOS: length of stay; ERM: empirical risk minimization; IRM: invariant risk minimization; AL: adversarial learning; GroupDRO: group distributionally robust optimization; CORAL: correlation alignment; MMD: maximum mean discrepancy; AUROC: area under the receiver operating characteristics curve; AUPRC: area under the pre…”
BackgroundInfluentialGuo, L. L., Pfohl, S. R., Fries, J., Posada, J., Fleming, S. L., Aftandilian, C., Shah, N., & Sung, L. (2021) · Applied Clinical Informatics · doi:10.1055/s-0041-1735184 · Cited by 96 · see citing works
Measure and Improve Robustness in NLP Models: A Survey
Wang, X., Wang, H., & Yang, D. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.339 · Cited by 90 · see citing works
In Search of Lost Domain Generalization
Gulrajani, I., & Lopez-Paz, D. (2020) · arXiv · doi:10.48550/arxiv.2007.01434 · Cited by 440 · see citing works
“Akuzawa et al. (2019) extend DANN by considering cases where there exists an statistical dependence between the domain and the class label variables. Albuquerque et al. (2019) extend DANN by considering one-versus-all adversaries that try to predict to which training domain does each of the examples belong to.”
BackgroundMethodInfluentialSelf-citationHow to remove or control confounds in predictive models, with applications to brain biomarkers
Chyzhyk, D., Varoquaux, G., Milham, M., & Thirion, B. (2022) · GigaScience · doi:10.1093/gigascience/giac014 · Cited by 65 · see citing works
Understanding the Failure Modes of Out-of-Distribution Generalization
Nagarajan, V., Andreassen, A., & Neyshabur, B. (2020) · arXiv · doi:10.48550/arxiv.2010.15775 · Cited by 55 · see citing works
PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domains
Ben-David, E., Oved, N., & Reichart, R. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00468 · Cited by 54 · see citing works
CausPref: Causal Preference Learning for Out-of-Distribution Recommendation
He, Y., Wang, Z., Cui, P., Zou, H., Zhang, Y., Cui, Q., & Jiang, Y. (2022) · Proceedings of the ACM Web Conference 2022 · doi:10.1145/3485447.3511969 · Cited by 50 · see citing works
Zhou, F., Mao, Y., Yu, L., Yang, Y., & Zhong, T. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.232 · Cited by 29 · see citing works
An Investigation of Critical Issues in Bias Mitigation Techniques
Shrestha, R., Kafle, K., & Kanan, C. (2022) · 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv51458.2022.00257 · Cited by 22 · see citing works
Causal Learning for Socially Responsible AI
Cheng, L., Mosallanezhad, A., Sheth, P., & Liu, H. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/598 · Cited by 15 · see citing works
Robust agents learn causal world models
Richens, J., & Everitt, T. (2024) · arXiv · doi:10.48550/arxiv.2402.10877 · Cited by 5 · see citing works
“We refer to σ v i as a domain indicator (Correa & Bareinboim, 2020) (it has also been called an environment index, Arjovsky et al., 2019).”
MethodBeyond Statistical Learning: Exact Learning Is Essential for General Intelligence
György, A., Lattimore, T., Lazić, N., & Szepesvári, C. (2025) · arXiv · doi:10.48550/arxiv.2506.23908
Zhang, J., & Bottou, L. (2025) · arXiv · doi:10.48550/arxiv.2507.03285
Self-citation
- Authors
- Arjovsky, M., Bottou, L., Gulrajani, I., & Lopez-Paz, D.
- ORCID
- Bottou, Léon
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1907.02893
- OpenAlex
- W2953494151
- Cited by
- 927
- References
- 6
- Citations by year
- 2026: 15 · 2025: 101 · 2024: 201 · 2023: 203 · 2022: 121 · 2021: 166 · 2020: 106 · 2019: 14
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- invariant risk minimization, invariance, out-of-distribution generalization, causal structure, representation learning, distribution shift robustness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
