The DOI matches its Crossref record; 38 of the 40 references it cites were found in a registry or an index.
Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology
Open accessAbstract
Stain variation is a phenomenon observed when distinct pathology laboratories stain tissue slides that exhibit similar but not identical color appearance. Due to this color shift between laboratories, convolutional neural networks (CNNs) trained with images from one lab often underperform on unseen images from the other lab. Several techniques have been proposed to reduce the generalization error, mainly grouped into two categories: stain color augmentation and stain color normalization. The former simulates a wide variety of realistic stain variations during training, producing stain-invariant CNNs. The latter aims to match training and test color distributions in order to reduce stain variation. For the first time, we compared some of these techniques and quantified their effect on CNN classification performance using a heterogeneous dataset of hematoxylin and eosin histopathology images from 4 organs and 9 pathology laboratories. Additionally, we propose a novel unsupervised method to perform stain color normalization using a neural network. Based on our experimental results, we provide practical guidelines on how to use stain color augmentation and stain color normalization in future computational pathology applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Techniques such as rotation, scaling, color normalization, or stain augmentation help simulate variability encountered in real-world slides [57].”
From Artificial Intelligence in Dermatopathology: An Update and Review of the Current Literature (2026) · cites this paper as [57]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In histopathology applications-studying tissue slides under a microscope-this variation can arise from sources like differences in the patient population or in slide staining and image acquisition (Veta et al., 2016; Komura and Ishikawa, 2018; Tellez et al., 2019).”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Tellez et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“To improve accessibility, we pre-process the slides into patches and balance the dataset so that each split has a 50/50 class balance, making average accuracy is a reasonable measure of performance (Veeling et al., 2018; Tellez et al., 2019).”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Tellez et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In histopathology applications-studying tissue slides under a microscope-this variation can arise from sources like differences in the patient population or in slide staining and image acquisition (Veta et al., 2016; Komura and Ishikawa, 2018; Tellez et al., 2019).”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Tellez et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“specialized stain augmentation methods can close the in- and out-of-distribution accuracy gap on a variant of the dataset based on the same underlying slides (Tellez et al., 2019)”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Tellez et al., 2019
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prior work has shown that differences in staining between hospitals are the primary source of variation in this dataset”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Tellez et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Prior work suggests that staining differences are the main source of variation between hospitals in similar datasets (Tellez et al., 2019).”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Tellez et al., 2019
The cited source says what the claim attributes to it.
In the citing paper
“These methods are reasonably effective at mitigating stain variation, at least in some contexts (Tellez et al., 2019; Miller et al., 2021)”
From WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020) · cites this paper as Tellez et al., 2019
In the citing paper
“Beyond technical challenges, regulatory frameworks for AI in oncology remain evolving, with significant gaps in liability assignment when AI-assisted decisions lead to adverse outcomes [124].”
From Artificial Intelligence in Clinical Oncology: From Productivity Enhancement to Creative Discovery (2025) · cites this paper as [124]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818
Labaca-Castro, R. (2023) · Springer eBooks · doi:10.1007/978-3-658-40442-0_9 · Cited by 21,119
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 20,507
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Statistical Comparisons of Classifiers over Multiple Data Sets
Demšar, J. (2006) · doi:10.5555/1248547.1248548 · Cited by 12,051
(2016) · Cited by 11,679
scikit-image: image processing in Python
(2014) · doi:10.7717/peerj.453 · Cited by 6,794
Striving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., and the CAMELYON16 Consortium, Hermsen, M., Manson, Q. F., et al. (2017) · JAMA · doi:10.1001/jama.2017.14585 · Cited by 3,293 · cited 5×
“This popular method has been used by numerous researchers in recent public chal-6 lenges (B´andi et al. (2019); Bejnordi et al. (2017)).”
BackgroundMethodInfluentialSelf-citationReinhard, E., Adhikhmin, M., Gooch, B., & Shirley, P. S. (2001) · IEEE Computer Graphics and Applications · doi:10.1109/38.946629 · Cited by 2,949
Best practices for convolutional neural networks applied to visual document analysis
Simard, P. Y., Steinkraus, D., & Platt, J. C. (2005) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2003.1227801 · Cited by 2,862
Deconvolution and Checkerboard Artifacts
Odena, A., Dumoulin, V., & Olah, C. (2016) · Distill · doi:10.23915/distill.00003 · Cited by 1,639
A method for normalizing histology slides for quantitative analysis
Macenko, M., Niethammer, M., Marron, J. S., Borland, D., Woosley, J. T., Guan, X., Schmitt, C., & Thomas, N. E. (2009) · 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro · doi:10.1109/isbi.2009.5193250 · Cited by 1,482
Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support
(2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67558-9 · Cited by 1,149
Gland segmentation in colon histology images: The glas challenge contest
Sirinukunwattana, K., Pluim, J. P. W., Chen, H., Qi, X., Heng, P., Guo, Y. B., Wang, L. Y., Matuszewski, B. J., Bruni, E., Sanchez, U., et al. (2016) · Medical Image Analysis · doi:10.1016/j.media.2016.08.008 · Cited by 1,014
Machine Learning Methods for Histopathological Image Analysis
Komura, D., & Ishikawa, S. (2018) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2018.01.001 · Cited by 937
AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images
Albarqouni, S., Baur, C., Achilles, F., Belagiannis, V., Demirci, S., & Navab, N. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528120 · Cited by 606
Multi-class texture analysis in colorectal cancer histology
Kather, J. N., Weis, C.-A., Bianconi, F., Melchers, S. M., Schad, L. R., Gaiser, T., Marx, A., & Zöllner, F. G. (2016) · Scientific Reports · doi:10.1038/srep27988 · Cited by 578
Bandi, P., Geessink, O., Manson, Q., Van Dijk, M., Balkenhol, M., Hermsen, M., Ehteshami Bejnordi, B., Lee, B., Paeng, K., Zhong, A., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2867350 · Cited by 569
Self-citationKhan, A. M., Rajpoot, N., Treanor, D. E., & Magee, D. (2014) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2303294 · Cited by 522
Detecting Cancer Metastases on Gigapixel Pathology Images
Liu, Y., Gadepalli, K., Norouzi, M., Dahl, G. E., Kohlberger, T., Boyko, A., Venugopalan, S., Timofeev, A., Nelson, P. Q., Corrado, G. S., et al. (2017) · arXiv · doi:10.48550/arxiv.1703.02442 · Cited by 506
Predicting breast tumor proliferation from whole-slide images: The TUPAC16 challenge
Veta, M., Heng, Y. J., Stathonikos, N., Bejnordi, B. E., de Beça, F. F., Wollmann, T., Rohr, K., Shah, M., Wang, D., Rousson, M., et al. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.02.012 · Cited by 309
Self-citationStain Specific Standardization of Whole-Slide Histopathological Images
Bejnordi, B. E., Litjens, G. J. S., Timofeeva, N., Otte‐Höller, I., Homeyer, A., Karssemeijer, N., & van der Laak, J. A. W. M. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2476509 · Cited by 291
Self-citationTellez, D., Balkenhol, M., Otte-Höller, I., van de Loo, R., Vogels, R., Bult, P., Wauters, C., Vreuls, W., Mol, S., Karssemeijer, N., et al. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2820199 · Cited by 288
Self-citationCiompi, F., Geessink, O. G. F., Bejnordi, B. E., de Souza, G. S., Baidoshvili, A., Litjens, G. J. S., van Ginneken, B., Nagtegaal, I. D., & van der Laak, J. A. W. M. (2017) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2017.7950492 · Cited by 218
Self-citationStain Normalization using Sparse AutoEncoders (StaNoSA): Application to digital pathology
Janowczyk, A. R., Basavanhally, A. N., & Madabhushi, A. (2016) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2016.05.003 · Cited by 211
Proceedings of the International Conference on Document Analysis and Recognition
Value, N. (2003) · Cited by 207
Bulten, W., Bándi, P., Hoven, J., van de Loo, R., Lotz, J., Weiss, N., van der Laak, J. A. W. M., van Ginneken, B., de Kaa, C. H., & Litjens, G. J. S. (2019) · Scientific Reports · doi:10.1038/s41598-018-37257-4 · Cited by 161
Self-citationStain normalization of histopathology images using generative adversarial networks
Zanjani, F. G., Zinger, S., Bejnordi, B. E., van der Laak, J. A. W. M., & de With, P. H. N. (2018) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2018.8363641 · Cited by 148
Self-citationMachine learning approaches to analyze histological images of tissues from radical prostatectomies
Gertych, A., Ing, N., Ma, Z., Fuchs, T. J., Salman, S., Mohanty, S. K., Bhele, S., Velásquez, A., Amin, M. B., & Knudsen, B. S. (2015) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2015.08.002 · Cited by 117
Neural Stain-Style Transfer Learning using GAN for Histopathological ImagesPreprint
Cho, H., Lim, S., Choi, G., & Min, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.08543 · Cited by 106
Colour in digital pathology: a review
Clarke, E. L., & Treanor, D. E. (2016) · Histopathology · doi:10.1111/his.13079 · Cited by 78
Context-Based Normalization of Histological Stains Using Deep Convolutional Features
Bug, D., Schneider, S., Grote, A., Oswald, E., Feuerhake, F., Schüler, J., & Merhof, D. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-67558-9_16 · Cited by 37
Exploring automatic prostate histopathology image gleason grading via local structure modeling
Wang, D., Foran, D. J., Ren, J., Zhong, H., Kim, I. Y., & Qi, X. (2015) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2015.7318936 · Cited by 34
Scalable Histopathological Image Analysis via Active Learning
Zhu, Y., Zhang, S., Liu, W., & Metaxas, D. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10443-0_47 · Cited by 21
Research works printed but not linked: 1
- Haeberli, 1994, Image processing by linear interpolation and extrapolation, IRIS Uni. Mag., 28, 8
Cited by
Sorted by citations
Deep learning in histopathology: the path to the clinic
van der Laak, J., Litjens, G., & Ciompi, F. (2021) · Nature Medicine · doi:10.1038/s41591-021-01343-4 · Cited by 902
Self-citationDeep neural network models for computational histopathology: A survey
Srinidhi, C. L., Ciga, O., & Martel, A. L. (2021) · Medical Image Analysis · doi:10.1016/j.media.2020.101813 · Cited by 749
Transformer-based unsupervised contrastive learning for histopathological image classification
Wang, X., Yang, S., Zhang, J., Wang, M., Zhang, J., Yang, W., Huang, J., & Han, X. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102559 · Cited by 751
Pan-cancer computational histopathology reveals mutations, tumor composition and prognosis
Fu, Y., Jung, A. W., Viñas, R., González, S., Vöhringer, H. S., Shmatko, A., Yates, L. R., Jimenez‐Liñan, M., Moore, L., & Gerstung, M. (2020) · Nature Cancer · doi:10.1038/s43018-020-0085-8 · Cited by 655
A Multi-Organ Nucleus Segmentation Challenge
Kumar, N., Verma, R., Anand, D., Zhou, Y., Onder, O. F., Tsougenis, E., Chen, H., Heng, P., Li, J., Hu, Z., et al. (2019) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2019.2947628 · Cited by 583
Metadata
- Authors
- Tellez, D., Litjens, G., Bándi, P., Bulten, W., Bokhorst, J.-M., Ciompi, F., & van der Laak, J.
- Affiliations
- Radboud University Medical Center (NL)
- Corresponding author
- Tellez, David
- Venue
- Medical Image Analysis
- Publisher
- Elsevier BV
- Published
- 2019-12
- Volume
- 58
- Pages
- 101544
- ISSN
- 1361-8415 (ISSN-L)
- OpenAlex
- W2969278648
- Cited by
- 649
- References
- 40
- Citations by year
- 2026: 142 · 2025: 108 · 2024: 80 · 2023: 92 · 2022: 89 · 2021: 81 · 2020: 42 · 2019: 14
- FWCI
- 25.50
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Medical Image Analysis (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- KWF Kankerbestrijding
- Radboud Universitair Medisch Centrum
- Horizon 2020
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Imaging for Blood Diseases
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
- Keywords
- computational pathology, convolutional neural networks, histopathological image classification
- MeSH terms
- Unsupervised Machine Learning, Color, Eosine Yellowish-(YS), Hematoxylin, Humans, Pathology, Clinical, Staining and Labeling, Neural Networks, Computer, Datasets as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
