The DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Measuring Domain Shift for Deep Learning in Histopathology
Open accessAbstract
The high capacity of neural networks allows fitting models to data with high precision, but makes generalization to unseen data a challenge. If a domain shift exists, i.e. differences in image statistics between training and test data, care needs to be taken to ensure reliable deployment in real-world scenarios. In digital pathology, domain shift can be manifested in differences between whole-slide images, introduced by for example differences in acquisition pipeline - between medical centers or over time. In order to harness the great potential presented by deep learning in histopathology, and ensure consistent model behavior, we need a deeper understanding of domain shift and its consequences, such that a model's predictions on new data can be trusted. This work focuses on the internal representation learned by trained convolutional neural networks, and shows how this can be used to formulate a novel measure - the representation shift - for quantifying the magnitude of model-specific domain shift. We perform a study on domain shift in tumor classification of hematoxylin and eosin stained images, by considering different datasets, models, and techniques for preparing data in order to reduce the domain shift. The results show how the proposed measure has a high correlation with drop in performance when testing a model across a large number of different types of domain shifts, and how it improves on existing techniques for measuring data shift and uncertainty. The proposed measure can reveal how sensitive a model is to domain variations, and can be used to detect new data that a model will have problems generalizing to. We see techniques for measuring, understanding and overcoming the domain shift as a crucial step towards reliable use of deep learning in the future clinical pathology applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
Labaca-Castro, R. (2023) · Machine Learning under Malware Attack · doi:10.1007/978-3-658-40442-0_9 · Cited by 19,272
GAN(Generative Adversarial Nets)
(2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,270
UMAP: Uniform Manifold Approximation and Projection
McInnes, L., Healy, J., Saul, N., & Großberger, L. (2018) · Journal of Open Source Software · doi:10.21105/joss.00861 · Cited by 9,826
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
McInnes, L., Healy, J., & Melville, J. (2018) · arXiv · doi:10.48550/arxiv.1802.03426 · Cited by 7,101
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061
Adversarial examples in the physical world
Kurakin, A., Goodfellow, I. J., & Bengio, S. (2018) · Artificial Intelligence Safety and Security · doi:10.1201/9781351251389-8 · Cited by 5,375
Adversarial Discriminative Domain Adaptation
Tzeng, E., Hoffman, J., Saenko, K., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.316 · Cited by 5,154
Learning Transferable Features with Deep Adaptation Networks
Long, M., Cao, Y., Wang, J., & Jordan, M. I. (2015) · arXiv · doi:10.48550/arxiv.1502.02791 · Cited by 4,505
Deep CORAL: Correlation Alignment for Deep Domain Adaptation
Sun, B., & Saenko, K. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_35 · Cited by 3,757
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435
Domain randomization for transferring deep neural networks from simulation to the real world
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., & Abbeel, P. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202133 · Cited by 3,124 · cited 1×
“Insummary,themaximumactivationfeaturespointtothe high sensitivity of training setup, where differences in model”
BackgroundTorralba, A., & Efros, A. A. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995347 · Cited by 2,440 · cited 1×
“If the unseen data has different distribution, i.e. a domain shift exists, the problem is significantly more difficult [1], [2] – even the smallest changes in the statistics as compared to the training data can cause a DNN to fail completely [3].”
BackgroundDeep domain confusion Maximizing for domain invariance
Tzeng, E., Hoffman, J., Zhang, N., Saenko, K., & Darrell, T. (2014) · arXiv · doi:10.48550/arxiv.1412.3474 · Cited by 2,285
Deep visual domain adaptation: A survey
Wang, M., & Deng, W. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.05.083 · Cited by 2,209
Analysis of Representations for Domain Adaptation
Ben-David, S., Blitzer, J., Crammer, K., & Pereira, F. (2007) · Advances in Neural Information Processing Systems 19 · doi:10.7551/mitpress/7503.003.0022 · Cited by 2,019
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., & Brendel, W. (2018) · arXiv · doi:10.48550/arxiv.1811.12231 · Cited by 1,490 · cited 1×
“…DNNs trained with natural images, for example what representations the model learns, how the choice of loss function and batch size affects the representation [38]– [40] and how this can be leveraged for transfer learning, novelty detection and detection of adversarial examples [39], [41], [42].”
BackgroundDomain Generalization with Adversarial Feature Learning
Li, H., Pan, S. J., Wang, S., & Kot, A. C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00566 · Cited by 1,213
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02530 · Cited by 989
Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization
Tremblay, J., Prakash, A., Acuna, D., Brophy, M., Jampani, V., Anil, C., To, T., Cameracci, E., Boochoon, S., & Birchfield, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00143 · Cited by 884
Olah, C., Mordvintsev, A., & Schubert, L. (2017) · Distill · doi:10.23915/distill.00007 · Cited by 880
Structure-Preserving Color Normalization and Sparse Stain Separation for Histological Images
Vahadane, A., Peng, T., Sethi, A., Albarqouni, S., Wang, L., Baust, M., Steiger, K., Schlitter, A. M., Esposito, I., & Navab, N. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2529665 · Cited by 825
Domain Generalization via Invariant Feature Representation
Muandet, K., Balduzzi, D., & Schölkopf, B. (2013) · arXiv · doi:10.48550/arxiv.1301.2115 · Cited by 771
Dataset Shift in Machine Learning
Adams, N. (2010) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2009.00624_10.x · Cited by 694
Domain Generalization for Object Recognition with Multi-task Autoencoders
Ghifary, M., Kleijn, W. B., Zhang, M., & Balduzzi, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.293 · Cited by 664
Tellez, D., Litjens, G., Bándi, P., Bulten, W., Bokhorst, J.-M., Ciompi, F., & van der Laak, J. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101544 · Cited by 649 · cited 1×
“Tellez et al. [37] investigate how much improvement color augmentation and/or stain normalization have on generalization performance to new datasets.”
BackgroundImproving the Robustness of Deep Neural Networks via Stability Training
Zheng, S., Song, Y., Leung, T., & Goodfellow, I. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.485 · Cited by 624
A Review of Domain Adaptation without Target Labels
Kouw, W. M., & Loog, M. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2945942 · Cited by 613
Why do deep convolutional networks generalize so poorly to small image transformations?
Azulay, A., & Weiss, Y. (2018) · arXiv · doi:10.48550/arxiv.1805.12177 · Cited by 428 · cited 1×
“Such perturbations can also be created without access to a trained model [3], and by performing natural image operations [9]–[11].”
Background1399 H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset
Litjens, G., Bandi, P., Ehteshami Bejnordi, B., Geessink, O., Balkenhol, M., Bult, P., Halilovic, A., Hermsen, M., van de Loo, R., Vogels, R., et al. (2018) · GigaScience · doi:10.1093/gigascience/giy065 · Cited by 418
Generalizing to Unseen Domains via Adversarial Data Augmentation
Volpi, R., Namkoong, H., Şener, O., Duchi, J. C., Murino, V., & Savarese, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.12018 · Cited by 343
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning
Papernot, N., & McDaniel, P. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.04765 · Cited by 335
Staingan: Stain Style Transfer for Digital Histological Images
Shaban, M. T., Baur, C., Navab, N., & Albarqouni, S. (2019) · 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019) · doi:10.1109/isbi.2019.8759152 · Cited by 327
(2017) · doi:10.48550/arxiv.1712.02779 · Cited by 300
Pearson’s Correlation Coefficient
(2008) · Encyclopedia of Public Health · doi:10.1007/978-1-4020-5614-7_2569 · Cited by 295
Adversarial Stain Transfer for Histopathology Image Analysis
Bentaieb, A., & Hamarneh, G. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2781228 · Cited by 231
Domain-Adversarial Neural Networks
Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., & Marchand, M. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.4446 · Cited by 221
Ciompi, F., Geessink, O., Bejnordi, B. E., de Souza, G. S., Baidoshvili, A., Litjens, G., van Ginneken, B., Nagtegaal, I., & van der Laak, J. (2017) · 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017) · doi:10.1109/isbi.2017.7950492 · Cited by 184
Domain-Adversarial Neural Networks to Address the Appearance Variability of Histopathology Images
Lafarge, M. W., Pluim, J. P. W., Eppenhof, K. A. J., Moeskops, P., & Veta, M. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67558-9_10 · Cited by 179
Stain normalization of histopathology images using generative adversarial networks
Zanjani, F. G., Zinger, S., Bejnordi, B. E., van der Laak, J. A. W. M., & de With, P. H. N. (2018) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2018.8363641 · Cited by 148
Color standardization and optimization in Whole Slide Imaging
Yagi, Y. (2011) · Diagnostic Pathology · doi:10.1186/1746-1596-6-s1-s15 · Cited by 123
(2008) · Springer eBooks · doi:10.1007/978-0-387-32833-1_214 · Cited by 114
Analyzing and Improving Representations with the Soft Nearest Neighbor LossPreprint
Frosst, N., Papernot, N., & Hinton, G. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.01889 · Cited by 112
A Closer Look at Domain Shift for Deep Learning in HistopathologyPreprint
Stacke, K., Eilertsen, G., Unger, J., & Lundström, C. F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11575 · Cited by 80
Self-citationTransfusion: Understanding Transfer Learning with Applications to Medical Imaging
Raghu, M., Zhang, C., Kleinberg, J. M., & Samy Bengio, A. (2019) · Cited by 73
Balkenhol, M. C. A., Karssemeijer, N., Litjens, G. J. S., van der Laak, J. A. W. M., Ciompi, F., & Tellez, D. (2018) · SPIE Medical Imaging: Digital and Computational Pathology · doi:10.1117/12.2293048 · Cited by 71
Unsupervised Domain Adaptation for Classification of Histopathology Whole-Slide Images
Ren, J., Hacihaliloglu, I., Singer, E. A., Foran, D. J., & Qi, X. (2019) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2019.00102 · Cited by 70
de Bel, T., Hermsen, M., Kers, J., van der Laak, J. A. W. M., & Litjens, G. J. S. (2018) · Data Archiving and Networked Services (DANS) · Cited by 60
Adversarial Domain Adaptation and Pseudo-Labeling for Cross-Modality Microscopy Image Quantification
Xing, F., Bennett, T. D., & Ghosh, D. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32239-7_82 · Cited by 43
Generalization of prostate cancer classification for multiple sites using deep learning
Arvidsson, I., Overgaard, N. C., Marginean, F.-E., Krzyzanowska, A., Bjartell, A. S., Åström, K., & Heyden, A. (2018) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2018.8363552 · Cited by 34
Finding the Unknown: Novelty Detection with Extreme Value Signatures of Deep Neural Activations
Schultheiß, A., Käding, C., Freytag, A., & Denzler, J. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-66709-6_19 · Cited by 20
Cited by
Domain adaptation for medical image analysis: A survey
Guan, H., & Liu, M. (2022) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2021.3117407 · Cited by 1,057
Deep neural network models for computational histopathology: A survey
Srinidhi, C. L., Ciga, O., & Martel, A. L. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101813 · Cited by 749
Deep learning-enabled virtual histological staining of biological samples
Bai, B., Yang, X., Li, Y., Zhang, Y., Pillar, N., & Ozcan, A. (2023) · Light: Science & Applications · doi:10.1038/s41377-023-01104-7 · Cited by 345
Designing deep learning studies in cancer diagnostics
Kleppe, A., Skrede, O.-J., De Raedt, S., Liestøl, K., Kerr, D. J., & Danielsen, H. E. (2021) · Nature Reviews Cancer · doi:10.1038/s41568-020-00327-9 · Cited by 316
Data drift in medical machine learning: implications and potential remedies
Sahiner, B., Chen, W., Samala, R. K., & Petrick, N. (2023) · The British Journal of Radiology · doi:10.1259/bjr.20220878 · Cited by 279
Metadata
- Authors
- Stacke, K., Eilertsen, G., Unger, J., & Lundstrom, C.
- Venue
- IEEE Journal of Biomedical and Health Informatics
- Published
- 2021-02
- Volume
- 25
- Issue
- 2
- Pages
- 325-336
- ISSN
- 2168-2194 (ISSN-L)
- 2168-2208
- OpenAlex
- W3094111018
- Cited by
- 318
- References
- 58
- FWCI
- 10.80
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.