CoChat
Sign Up
PreprintCited by 55Multimodal Machine Learning ApplicationsAI in cancer detectionImage Retrieval and Classification Techniques

Quilt-1M: One Million Image-Text Pairs for Histopathology

Open access
Ikezogwo, Wisdom Oluchi, Seyfioglu, Mehmet Saygin, Ghezloo, Fatemeh, Geva, Dylan Stefan Chan, Mohammed, Fatwir Sheikh, Anand, Pavan Kumar, Krishna, Ranjay, Shapiro, Linda·2023·arXiv
DOI 10.48550/arxiv.2306.11207arXiv 2306.11207PMC PMC11090501
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent accelerations in multi-modal applications have been made possible with the plethora of image and text data available online. However, the scarcity of analogous data in the medical field, specifically in histopathology, has slowed comparable progress. To enable similar representation learning for histopathology, we turn to YouTube, an untapped resource of videos, offering $1,087$ hours of valuable educational histopathology videos from expert clinicians. From YouTube, we curate QUILT: a large-scale vision-language dataset consisting of $802, 144$ image and text pairs. QUILT was automatically curated using a mixture of models, including large language models, handcrafted algorithms, human knowledge databases, and automatic speech recognition. In comparison, the most comprehensive datasets curated for histopathology amass only around $200$K samples. We combine QUILT with datasets from other sources, including Twitter, research papers, and the internet in general, to create an even larger dataset: QUILT-1M, with $1$M paired image-text samples, marking it as the largest vision-language histopathology dataset to date. We demonstrate the value of QUILT-1M by fine-tuning a pre-trained CLIP model. Our model outperforms state-of-the-art models on both zero-shot and linear probing tasks for classifying new histopathology images across $13$ diverse patch-level datasets of $8$ different sub-pathologies and cross-modal retrieval tasks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.11207

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 61

  1. Agrawal M, Hegselmann S, Lang H, Kim Y, and Sontag D. Large language models are zero-shot clinical information extractors. arXiv preprint arXiv:2205.12689, 2022. · arXiv
  2. Amith M, Cui L, Roberts K, Xu H, and Tao C. Ontology of consumer health vocabulary: providing a formal and interoperable semantic resource for linking lay language and medical terminology. In 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 1177-1178. IEEE, 2019.
  3. Araujo A, Chaves J, Lakshman H, Angst R, and Girod B. Large-scale query-by-image video retrieval using bloom filters. arXiv preprint arXiv:1604.07939, 2016. · arXiv
  4. Aresta G, Araújo T, Kwok S, Chennamsetty SS, Safwan M, Alex V, Marami B, Prastawa M, Chan M, Donovan M, et al. Bach: Grand challenge on breast cancer histology images. Medical image analysis, 56:122-139, 2019. [PubMed: 31226662]
  5. Arunachalam HB, Mishra R, Daescu O, Cederberg K, Rakheja D, Sengupta A, Leonard D, Hallac R, and Leavey P. Viable and necrotic tumor assessment from whole slide images of osteosarcoma using machine-learning and deep-learning models. PloS one, 14(4):e0210706, 2019.
  6. Baker B, Akkaya I, Zhokov P, Huizinga J, Tang J, Ecoffet A, Houghton B, Sampedro R, and Clune J. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-24654, 2022.
  7. Borkowski AA, Bui MM, Thomas LB, Wilson CP, DeLand LA, and Mastorides SM. Lung and colon cancer histopathological image dataset (lc25000). arXiv preprint arXiv:1912.12142, 2019. · arXiv
  8. Brummer O, Polonen P, Mustjoki S, and Bruck O. Integrative analysis of histological textures and lymphocyte infiltration in renal cell carcinoma using deep learning. bioRxiv, pages 2022-08, 2022.
  9. Caron M, Touvron H, Misra I, Jégou H, Mairal J, Bojanowski P, and Joulin A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
  10. Chen RJ, Lu MY, Weng W-H, Chen TY, Williamson DF, Manz T, Shady M, and Mahmood F. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4025, 2021.
  11. Chen RJ, Chen C, Li Y, Chen TY, Trister AD, Krishnan RG, and Mahmood F. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16144-16155, 2022.
  12. Chen T, Kornblith S, Norouzi M, and Hinton G. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
  13. Ding B, Qin C, Liu L, Bing L, Joty S, and Li B. Is gpt-3 a good data annotator? arXiv preprint arXiv:2212.10450, 2022. · arXiv
  14. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. · arXiv
  15. Doukhan D, Carrive J, Vallet F, Larcher A, and Meignier S. An open-source speaker gender detection framework for monitoring gender equality. In Acoustics Speech and Signal Processing (ICASSP), 2018 IEEE International Conference on. IEEE, 2018.
  16. Eslami S, de Melo G, and Meinel C. Does clip benefit visual question answering in the medical domain as much as it does in the general domain? arXiv preprint arXiv:2112.13906, 2021. · arXiv
  17. Fragoso G, de Coronado S, Haber M, Hartel F, and Wright L. Overview and utilization of the nci thesaurus. Comparative and functional genomics, 5(8):648-654, 2004. [PubMed: 18629178]
  18. Gamper J and Rajpoot N. Multiple instance captioning: Learning representations from histopathology textbooks and articles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16549-16559, 2021.
  19. He X, Zhang Y, Mou L, Xing E, and Xie P. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286, 2020. · arXiv
  20. Huang S-C, Shen L, Lungren MP, and Yeung S. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942-3951, 2021.
  21. Huang Z, Bianchi F, Yuksekgonul M, Montine T, and Zou J. Leveraging medical twitter to build a visual-language foundation model for pathology ai. bioRxiv, pages 2023-03, 2023.
  22. Ikezogwo WO, Seyfioglu MS, and Shapiro L. Multi-modal masked autoencoders learn compositional histopathological representations. arXiv preprint arXiv:2209.01534, 2022. · arXiv
  23. Jobin K, Mondal A, and Jawahar C. Docfigure: A dataset for scientific document figure classification. In 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 1, pages 74-79. IEEE, 2019.
  24. Johnson AE, Pollard TJ, Greenbaum NR, Lungren MP, Deng C.-y., Peng Y, Lu Z, Mark RG, Berkowitz SJ, and Horng S. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042, 2019. · arXiv
  25. Jupp S, Malone J, Burdett T, Heriche J-K, Williams E, Ellenberg J, Parkinson H, and Rustici G. The cellular microscopy phenotype ontology. Journal of biomedical semantics, 7: 1-8, 2016. [PubMed: 26759709]
  26. Kembhavi A, Salvato M, Kolve E, Seo M, Hajishirzi H, and Farhadi A. A diagram is worth a dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 235-251. Springer, 2016.
  27. Kojima T, Gu SS, Reid M, Matsuo Y, and Iwasawa Y. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022. · arXiv
  28. Liu J, Wang Q, Fan H, Wang S, Li W, Tang Y, Wang D, Zhou M, and Chen L. Automatic label correction for the accurate edge detection of overlapping cervical cells. arXiv preprint arXiv:2010.01919, 2020. · arXiv
  29. Liu S, Zhu C, Xu F, Jia X, Shi Z, and Jin M. Bci: Breast cancer immunohistochemical image generation through pyramid pix2pix. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1815-1824, 2022.
  30. Liu Z, Luo P, Wang X, and Tang X. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
  31. Liu Z, Mao H, Wu C-Y, Feichtenhofer C, Darrell T, and Xie S. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976-11986, 2022.
  32. Loshchilov I and Hutter F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. · arXiv
  33. Marini N, Marchesin S, Otálora S, Wodzinski M, Caputo A, Van Rijthoven M, Aswolinskiy W, Bokhorst J-M, Podareanu D, Petters E, et al. Unleashing the potential of digital pathology data by training computer-aided diagnosis models without human annotations. NPJ digital medicine, 5(1):102, 2022. [PubMed: 35869179]
  34. Morris D, Müller-Budack E, and Ewerth R. Slideimages: a dataset for educational image classification. In Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14-17, 2020, Proceedings, Part II 42, pages 289-296. Springer, 2020.
  35. Neumann M, King D, Beltagy I, and Ammar W. ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 319-327, Florence, Italy, Aug. 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-5034. URL https://www.aclweb.org/anthology/W19-5034. · DOI
  36. Noy NF, Musen MA, Mejino JL Jr, and Rosse C. Pushing the envelope: challenges in a frame- based representation of human anatomy. Data & Knowledge Engineering, 48(3): 335-359, 2004.
  37. Pelka O, Koitka S, Rückert J, Nensa F, and Friedrich CM. Radiology objects in context (roco): a multimodal image dataset. In Intravascular Imaging and Computer Assisted Stenting and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis: 7th Joint International Workshop, CVII-STENT 2018 and Third International Workshop, LABELS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 3, pages 180-189. Springer, 2018.
  38. Pont-Tuset J, Uijlings J, Changpinyo S, Soricut R, and Ferrari V. Connecting vision and language with localized narratives. In ECCV, 2020.
  39. Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  40. Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
  41. Radford A, Kim JW, Xu T, Brockman G, McLeavey C, and Sutskever I. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356, 2022. · arXiv
  42. Rose S, Engel D, Cramer N, and Cowley W. Automatic keyword extraction from individual documents. Text mining: applications and theory, pages 1-20, 2010.
  43. Santos T, Tariq A, Das S, Vayalpati K, Smith GH, Trivedi H, and Banerjee I. Pathologybert- pre-trained vs. a new transformer language model for pathology domain. arXiv preprint arXiv:2205.06885, 2022. · arXiv
  44. Schofield PN, Sundberg JP, Sundberg BA, McKerlie C, and Gkoutos GV. The mouse pathology ontology, mpath; structure and applications. Journal of biomedical semantics, 4(1): 1-8, 2013. [PubMed: 23286462]
  45. Schuhmann C, Beaumont R, Vencu R, Gordon C, Wightman R, Cherti M, Coombes T, Katta A, Mullis C, Wortsman M, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022. · arXiv
  46. Selvaraju R, Cogswell M, Das A, Vedantam R, Parikh D, and Batra D. Grad-cam: Visual explanations from deep networks via gradient-based localization. arxiv 2016. arXiv preprint arXiv:1610.02391. · arXiv
  47. Singh A, Natarajan V, Shah M, Jiang Y, Chen X, Batra D, Parikh D, and Rohrbach M. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
  48. Singh H and Graber ML. Improving diagnosis in health care-the next imperative for patient safety. The New England journal of medicine, 373(26):2493-2495, 2015. [PubMed: 26559457]
  49. Singhal K, Azizi S, Tu T, Mahdavi SS, Wei J, Chung HW, Scales N, Tanwani A, Cole- Lewis H, Pfohl S, et al. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138, 2022. · arXiv
  50. Veeling BS, Linmans J, Winkens J, Cohen T, and Welling M. Rotation equivariant cnns for digital pathology. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, pages 210-218. Springer, 2018.
  51. Voigtlaender P, Changpinyo S, Pont-Tuset J, Soricut R, and Ferrari V. Connecting Vision and Language with Video Localized Narratives. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
  52. Wang S, Liu Y, Xu Y, Zhu C, and Zeng M. Want to reduce labeling cost? gpt-3 can help. arXiv preprint arXiv:2108.13487, 2021. · arXiv
  53. Wei J, Suriawinata A, Ren B, Liu X, Lisovsky M, Vaickus L, Brown C, Baker M, Tomita N, Torresani L, et al. A petri dish for histopathology image analysis. In Artificial Intelligence in Medicine: 19th International Conference on Artificial Intelligence in Medicine, AIME 2021, Virtual Event, June 15-18, 2021, Proceedings, pages 11-24. Springer, 2021.
  54. Weitz P, Valkonen M, Solorzano L, Carr C, Kartasalo K, Boissin C, Koivukoski S, Kuusela A, Rasic D, Feng Y, et al. Acrobat-a multi-stain breast cancer histological whole-slide-image data set from routine diagnostics for computational pathology. arXiv preprint arXiv:2211.13621, 2022. · arXiv
  55. Wu H, Wang W, Wan Y, Jiao W, and Lyu M. Chatgpt or grammarly? evaluating chatgpt on grammatical error correction benchmark. arXiv preprint arXiv:2303.13648, 2023. · arXiv
  56. Wu W, Mehta S, Nofallah S, Knezevich S, May CJ, Chang OH, Elmore JG, and Shapiro LG. Scale-aware transformers for diagnosing melanocytic lesions. IEEE Access, 9: 163526-163541, 2021. [PubMed: 35211363]
  57. Zellers R, Lu X, Hessel J, Yu Y, Park JS, Cao J, Farhadi A, and Choi Y. Merlot: Multimodal neural script knowledge models. Advances in Neural Information Processing Systems, 34:23634- 23651, 2021.
  58. Zellers R, Lu J, Lu X, Yu Y, Zhao Y, Salehi M, Kusupati A, Hessel J, Farhadi A, and Choi Y. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16375-16387, 2022.
  59. Zhang S, Xu Y, Usuyama N, Bagga J, Tinn R, Preston S, Rao R, Wei M, Valluri N, Wong C, et al. Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915, 2023. · arXiv
  60. Zhang Y, Jiang H, Miura Y, Manning CD, and Langlotz CP. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pages 2-25. PMLR, 2022.
  61. Zhou B, Lapedriza A, Khosla A, Oliva A, and Torralba A. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.

Cited by

Sorted by citations

Authors
Ikezogwo, W. O., Seyfioglu, M. S., Ghezloo, F., Geva, D. S. C., Mohammed, F. S., Anand, P. K., Krishna, R., & Shapiro, L.
Affiliations
University of Washington (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Volume
36
Issue
DB1
Pages
37995-38017
DOI
10.48550/arxiv.2306.11207
arXiv
2306.11207
PMC
PMC11090501
OpenAlex
W4381573401
Cited by
55
References
2
Citations by year
2026: 3 · 2025: 27 · 2024: 24 · 2023: 1
FWCI
4.98
Type
journal article
Language
en
Open access
Licence
Locations
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion, other-oa)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Cancer Institute (R01CA201376, R01CA151306, R01CA225585, R01CA200690)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
Keywords
histopathology, image-text pairs, YouTube educational videos, large language models, CLIP fine-tuning, zero-shot classification, linear probing, cross-modal retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review