CoChat
Sign Up
PreprintCited by 27AI in cancer detectionArtificial Intelligence in Healthcare and EducationRadiomics and Machine Learning in Medical Imaging

RudolfV: A Foundation Model by Pathologists for Pathologists

Open access
Dippel, Jonas, Feulner, Barbara, Winterhoff, Tobias, Milbich, Timo, Tietz, Stephan, Schallenberg, Simon, Dernbach, Gabriel, Kunft, Andreas +12 more·2024·arXiv
DOI 10.48550/arxiv.2401.04079arXiv 2401.04079
Open this paper in CoChat — free accountOpen PDF

Abstract

Artificial intelligence has started to transform histopathology impacting clinical diagnostics and biomedical research. However, while many computational pathology approaches have been proposed, most current AI models are limited with respect to generalization, application variety, and handling rare diseases. Recent efforts introduced self-supervised foundation models to address these challenges, yet existing approaches do not leverage pathologist knowledge by design. In this study, we present a novel approach to designing foundation models for computational pathology, incorporating pathologist expertise, semi-automated data curation, and a diverse dataset from over 15 laboratories, including 58 tissue types, and encompassing 129 different histochemical and immunohistochemical staining modalities. We demonstrate that our model "RudolfV" surpasses existing state-of-the-art foundation models across different benchmarks focused on tumor microenvironment profiling, biomarker evaluation, and reference case search while exhibiting favorable robustness properties. Our study shows how domain-specific knowledge can increase the efficiency and performance of pathology foundation models and enable novel application areas.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2401.04079

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Virchow is the largest foundation model for computational pathology to date.

    “we present Virchow, the largest foundation model for computational pathology to date.”

    From A foundation model for clinical-grade computational pathology and rare cancers detection (2024) · cites this paper as [40]

  • These works demonstrate that image features produced with self-supervised learning of pathology images outperform image features trained on natural images and that performance improves with scale.

    “These works demonstrate that image features produced with self-supervised learning of pathology images outperform image features trained on natural images and that performance improves with scale.”

    From A foundation model for clinical-grade computational pathology and rare cancers detection (2024) · cites this paper as [40]

  • Recent pioneering works have utilized datasets ranging from 30,000 to 400,000 WSIs to train foundation models ranging in size from 28 million to 307 million parameters.

    “Despite the challenges in collecting large-scale datasets in the pathology domain, recent pioneering works have utilized datasets ranging from 30,000 to 400,000 WSIs to train foundation models ranging in size from 28 million to 307 million parameters35–42”

    From A foundation model for clinical-grade computational pathology and rare cancers detection (2024) · cites this paper as [40]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 54

  1. F. Klauschen et al., "Toward explainable artificial intelli- gence for precision pathology," Annual Review of Pathol- ogy: Mechanisms of Disease, vol. 19, pp. 541-570, 2024. doi: 10.1146/annurev-pathmechdis-051222-113147. · DOI
  2. J. A. Diao et al., "Human-interpretable image features de- rived from densely mapped cancer pathology slides pre- dict diverse molecular phenotypes," Nature communica- tions, vol. 12, no. 1, p. 1613, 2021.
  3. P. Raciti et al., "Clinical validation of artificial intelligence-augmented pathology diagnosis demonstrates significant gains in diagnostic accuracy in prostate cancer detection," Archives of Pathology & Laboratory Medicine, vol. 147, no. 10, pp. 1178-1185, 2023.
  4. A. Binder et al., "Morphological and molecular breast can- cer profiling through explainable machine learning," Na- ture Machine Intelligence, vol. 3, no. 4, pp. 355-366, 2021.
  5. P. Keyl et al., "Patient-level proteomic network prediction by explainable artificial intelligence," NPJ Precision On- cology, vol. 6, no. 1, p. 35, 2022.
  6. R. Bommasani et al., "On the opportunities and risks of foundation models," arXiv preprint arXiv:2108.07258, 2021. · arXiv
  7. T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A sim- ple framework for contrastive learning of visual represen- tations," in International conference on machine learning, PMLR, 2020, pp. 1597-1607.
  8. M. Oquab et al., "Dinov2: Learning robust visual features without supervision," arXiv preprint arXiv:2304.07193, 2023. · arXiv
  9. K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, "Masked autoencoders are scalable vision learners," in Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16 000-16 009.
  10. R. Krishnan, P. Rajpurkar, and E. J. Topol, "Self- supervised learning in medicine and healthcare," Nature Biomedical Engineering, vol. 6, no. 12, pp. 1346-1352, 2022.
  11. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., "Language models are unsupervised mul- titask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.
  12. T. Brown et al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
  13. OpenAI, "Gpt-4 technical report," arXiv preprint arXiv:2303.08774, 2023. · arXiv
  14. T. Tu et al., "Towards generalist biomedical ai," arXiv preprint arXiv:2307.14334, 2023. · arXiv
  15. O. Ciga, T. Xu, and A. L. Martel, "Self supervised contrastive learning for digital histopathology," Machine Learning with Applications, vol. 7, p. 100 198, 2022.
  16. R. J. Chen et al., "Scaling vision transformers to gigapixel images via hierarchical self-supervised learning," in Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, 2022, pp. 16 144-16 155.
  17. X. Wang et al., "Transformer-based unsupervised con- trastive learning for histopathological image classification," Medical image analysis, vol. 81, p. 102 559, 2022.
  18. A. Filiot et al., "Scaling self-supervised learning for histopathology with masked image modeling," medRxiv, pp. 2023-07, 2023.
  19. E. Vorontsov et al., "Virchow: A million-slide dig- ital pathology foundation model," arXiv preprint arXiv:2309.07778, 2023. · arXiv
  20. G. Shaikovski et al., "Prism: A multi-modal generative foundation model for slide-level histopathology," arXiv preprint arXiv:2405.10254, 2024. · arXiv
  21. H. Xu et al., "A whole-slide foundation model for digital pathology from real-world data," Nature, pp. 1-8, 2024.
  22. D. M. Reese, "Fundamentals-rudolf virchow and modern medicine.," Western journal of medicine, vol. 169, no. 2, p. 105, 1998.
  23. M. Schultz, "Rudolf virchow," Emerging infectious dis- eases, vol. 14, no. 9, p. 1480, 2008.
  24. J. Zhou et al., "Ibot: Image bert pre-training with online tokenizer," arXiv preprint arXiv:2111.07832, 2021. · arXiv
  25. L. Cords et al., "Cancer-associated fibroblast phenotypes are associated with patient outcome in non-small cell lung cancer," en, Cancer Cell, vol. 42, no. 3, 396-412.e5, Mar. 2024.
  26. L. Keren et al., "A structured tumor-immune microenvi- ronment in triple negative breast cancer revealed by mul- tiplexed ion beam imaging," en, Cell, vol. 174, no. 6, 1373- 1387.e19, Sep. 2018.
  27. S. Kalra et al., "Yottixel-an image search engine for large archives of histopathology whole slide images," Medical Image Analysis, vol. 65, p. 101 757, 2020.
  28. C. Chen, M. Y. Lu, D. F. Williamson, T. Y. Chen, A. J. Schaumberg, and F. Mahmood, "Fast and scalable search of whole-slide images via self-supervised deep learning," Nature biomedical engineering, vol. 6, no. 12, pp. 1420- 1434, 2022.
  29. X. Wang et al., "Retccl: Clustering-guided contrastive learning for whole-slide image retrieval," Medical image analysis, vol. 83, p. 102 645, 2023.
  30. S. Li, Y. Zhao, J. Zhang, T. Yu, J. Zhang, and Y. Gao, "High-order correlation-guided slide-level histology retrieval with self-supervised hashing," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
  31. J. Shlens, A tutorial on principal component analysis, 2014. [Online]. Available: http://arxiv.org/abs/1404.1100. · arXiv
  32. Y. Bengio, A. Courville, and P. Vincent, "Representation learning: A review and new perspectives," IEEE Trans- actions on Pattern Analysis and Machine Intelligence, vol. 35, no. 8, pp. 1798-1828, 2013.
  33. H. V. Vo et al., Automatic data curation for self- supervised learning: A clustering-based approach, 2024. arXiv: 2405.15613 [cs.LG]. · arXiv
  34. M. Assran et al., "Self-supervised learning from images with a joint-embedding predictive architecture," in Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, 2023, pp. 15 619-15 629.
  35. D. Juyal et al., Pluto: Pathology-universal transformer, 2024. arXiv: 2405.07905 [eess.IV]. · arXiv
  36. A. Dosovitskiy et al., "An image is worth 16x16 words: Transformers for image recognition at scale," in Interna- tional Conference on Learning Representations, 2020.
  37. L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang, Vision mamba: Efficient visual representation learning with bidirectional state space model, 2024. arXiv: 2401.09417 [cs.CV]. · arXiv
  38. Y. Yue and Z. Li, Medmamba: Vision mamba for medical image classification, 2024. arXiv: 2403.03849 [eess.IV]. · arXiv
  39. W. Wang et al., When an image is worth 1,024 x 1,024 words: A case study in computational pathology, 2023. arXiv: 2312.03558 [cs.CV]. · arXiv
  40. M. Tran et al., "Generating highly accurate pathology re- ports from gigapixel whole slide images with histogpt," medRxiv, 2024. doi: 10.1101/2024.03.15.24304211. · DOI
  41. A. C. Ruifrok, D. A. Johnston, et al., "Quantification of histochemical staining by color deconvolution," Analyti- cal and quantitative cytology and histology, vol. 23, no. 4, pp. 291-299, 2001.
  42. S. Lapuschkin, S. Wäldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller, "Unmasking clever hans pre- dictors and assessing what machines really learn," Nature communications, vol. 10, no. 1, p. 1096, 2019.
  43. E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley, "Color transfer between images," IEEE Computer graphics and applications, vol. 21, no. 5, pp. 34-41, 2001.
  44. M. Kang, H. Song, S. Park, D. Yoo, and S. Pereira, "Benchmarking self-supervised learning on diverse pathol- ogy datasets," in Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2023, pp. 3344-3354.
  45. S. Azizi et al., "Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging," Nature Biomedical Engineering, pp. 1-24, 2023.
  46. D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in 3rd International Conference on Learn- ing Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  47. A. Kirillov et al., "Segment anything," in Proceedings of the IEEE/CVF International Conference on Computer Vi- sion, 2023, pp. 4015-4026.
  48. J. Snell, K. Swersky, and R. Zemel, "Prototypical net- works for few-shot learning," Advances in neural informa- tion processing systems, vol. 30, 2017.
  49. Y. Wang, W.-L. Chao, K. Q. Weinberger, and L. Van Der Maaten, "Simpleshot: Revisiting nearest-neighbor classification for few-shot learning," arXiv preprint arXiv:1911.04623, 2019. · arXiv
  50. B. S. Veeling, J. Linmans, J. Winkens, T. Cohen, and M. Welling, "Rotation equivariant CNNs for digital pathol- ogy," Jun. 2018. arXiv: 1806.03962 [cs.CV]. · arXiv
  51. B. E. Bejnordi et al., "Diagnostic assessment of deep learn- ing algorithms for detection of lymph node metastases in women with breast cancer," Jama, vol. 318, no. 22, pp. 2199-2210, 2017.
  52. J. Wei et al., "A petri dish for histopathology image anal- ysis," in Artificial Intelligence in Medicine: 19th Interna- tional Conference on Artificial Intelligence in Medicine, AIME 2021, Virtual Event, June 15-18, 2021, Proceed- ings, Springer, 2021, pp. 11-24.
  53. J. N. Kather et al., "Predicting survival from colorectal cancer histology slides using deep learning: A retrospec- tive multicenter study," PLoS medicine, vol. 16, no. 1, e1002730, 2019.
  54. J. N. Kather et al., "Deep learning can predict microsatel- lite instability directly from histology in gastrointestinal cancer," Nature medicine, vol. 25, no. 7, pp. 1054-1056, 2019.

Cited by

Sorted by citations

Related

Authors
Dippel, J., Feulner, B., Winterhoff, T., Milbich, T., Tietz, S., Schallenberg, S., Dernbach, G., Kunft, A., Heinke, S., Eich, M.-L., Ribbat-Idel, J., Krupar, R., Anders, P., Prenißl, N., Jurmeister, P., Horst, D., Ruff, L., Müller, K.-R., Klauschen, F., & Alber, M.
Affiliations
  • Charité – Universitätsmedizin Berlin (DE)
  • German Cancer Research Center (DE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2401.04079
arXiv
2401.04079
OpenAlex
W4390723185
Cited by
27
References
0
Citations by year
2026: 1 · 2025: 18 · 2024: 8
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Funders
  • Bundesministerium für Bildung und Forschung (01 IS 14013 A, 031L0207D, 01IS14013A-E, 01GQ0850, 01IS18037A, 01GQ1115, 01IS18025A)
  • Korea University (2019-0-00079, 2022-0-00984)
  • Ministry of Science and ICT, South Korea (2022-0- 00984, 01GQ1115, 01GQ0850, 01IS18037A, 01IS18025A, 2019-0-00079)
  • Institute for Information and Communications Technology Promotion (2019–0–00079, 2022-, 2022-0-00984)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
foundation models, computational pathology, histopathology, self-supervised learning, biomarker analysis, immunohistochemistry, histochemical staining, rare diseases, model robustness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review