Diagnosing Failures of Fairness Transfer Across Distribution Shift in Real-World Medical Settings
Open accessThe DOI matches its OpenAlex record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Diagnosing and mitigating changes in model fairness under distribution shift is an important component of the safe deployment of machine learning in healthcare settings. Importantly, the success of any mitigation strategy strongly depends on the structure of the shift. Despite this, there has been little discussion of how to empirically assess the structure of a distribution shift that one is encountering in practice. In this work, we adopt a causal framing to motivate conditional independence tests as a key tool for characterizing distribution shifts. Using our approach in two medical applications, we show that this knowledge can help diagnose failures of fairness transfer, including cases where real-world shifts are more complex than is often assumed in the literature. Based on these results, we discuss potential remedies at each step of the machine learning pipeline.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Improving predictive inference under covariate shift by weighting the log-likelihood function
Shimodaira, H. (2000) · Journal of Statistical Planning and Inference · doi:10.1016/s0378-3758(00)00115-4 · Cited by 1,678
A Reductions Approach to Fair Classification
Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1803.02453 · Cited by 1,288
Addressing extreme propensity scores via the overlap weights
Li, F., & Thomas, L. E. (2018) · American Journal of Epidemiology · doi:10.1093/aje/kwy201 · Cited by 694
Wiens, J., Guttag, J., & Horvitz, E. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2013-002162 · Cited by 123
Residual unfairness in fair machine learning from prejudiced data
Kallus, N., & Zhou, A. (2018) · arXiv · doi:10.48550/arxiv.1806.02887 · Cited by 83
Cited by
Algorithmic fairness in artificial intelligence for medicine and healthcare
Chen, R. J., Wang, J. J., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Lu, M. Y., Sahai, S., & Mahmood, F. (2023) · Nature Biomedical Engineering · doi:10.1038/s41551-023-01056-8 · Cited by 750
The limits of fair medical imaging AI in real-world generalization
Yang, Y., Zhang, H., Gichoya, J. W., Katabi, D., & Ghassemi, M. (2024) · Nature Medicine · doi:10.1038/s41591-024-03113-4 · Cited by 285
“Thus, improving ID fairness may not lead to improvements in OOD fairness, highlighting the complex interplay between fairness and distribution shift 45,46 .”
Generative models improve fairness of medical classifiers under distribution shifts
Ktena, I., Wiles, O., Albuquerque, I., Rebuffi, S.-A., Tanno, R., Roy, A. G., Azizi, S., Belgrave, D., Kohli, P., Cemgil, T., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02838-6 · Cited by 165
“This is impressive as Schrouff et al. 18 demonstrated that improving fairness on in-distribution datasets does not guarantee performance improvements on OOD datasets.”
Self-citationGenerative models improve fairness of medical classifiers under distribution shifts
Wiles, O., Albuquerque, I., Rebuffi, S.-A., Tanno, R., Roy, A. G., Azizi, S., Belgrave, D., Kohli, P., Cemgil, T., Karthikesalingam, A., et al. (2023) · doi:10.21203/rs.3.rs-2976332/v1 · Cited by 165
Self-citationFair Machine Learning in Healthcare: A Survey
Feng, Q., Du, M., Zou, N., & Hu, X. (2025) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2024.3361836 · Cited by 12
Metadata
- Authors
- Schrouff, J., Harris, N., Koyejo, S., Alabdulmohsin, I., Schnider, E., Opsahl-Ong, K., Brown, A. P. Y., Roy, S., Mincu, D., Chen, C., Dieng, A., Liu, Y., Natarajan, V., Karthikesalingam, A., Heller, K., Chiappa, S., & D’Amour, A.
- Venue
- neural information processing systems
- Published
- 2022-01-01
- Pages
- 19304-19318
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133229083
- Cited by
- 35
- References
- 8
- FWCI
- 7.19
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.