CoChat
Sign Up
PreprintCited by 118Medical Image Segmentation TechniquesAdvanced Neural Network ApplicationsMultimodal Machine Learning Applications

BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once

Open access
Zhao, Theodore, Gu, Yu, Yang, Jianwei, Usuyama, Naoto, Lee, Ho Hin, Naumann, Tristan, Gao, Jianfeng, Crabtree, Angela +7 more·2024·arXiv
DOI 10.48550/arxiv.2405.12971arXiv 2405.12971
Open this paper in CoChat — free accountOpen PDF

Abstract

Biomedical image analysis is fundamental for biomedical discovery in cell biology, pathology, radiology, and many other biomedical domains. Holistic image analysis comprises interdependent subtasks such as segmentation, detection, and recognition of relevant objects. Here, we propose BiomedParse, a biomedical foundation model for imaging parsing that can jointly conduct segmentation, detection, and recognition for 82 object types across 9 imaging modalities. Through joint learning, we can improve accuracy for individual tasks and enable novel applications such as segmenting all relevant objects in an image through a text prompt, rather than requiring users to laboriously specify the bounding box for each object. We leveraged readily available natural-language labels or descriptions accompanying those datasets and use GPT-4 to harmonize the noisy, unstructured text information with established biomedical object ontologies. We created a large dataset comprising over six million triples of image, segmentation mask, and textual description. On image segmentation, we showed that BiomedParse is broadly applicable, outperforming state-of-the-art methods on 102,855 test image-mask-label triples across 9 imaging modalities (everything). On object detection, which aims to locate a specific object of interest, BiomedParse again attained state-of-the-art performance, especially on objects with irregular shapes (everywhere). On object recognition, which aims to identify all objects in a given image along with their semantic types, we showed that BiomedParse can simultaneously segment and label all biomedical objects in an image (all at once). In summary, BiomedParse is an all-in-one tool for biomedical image analysis by jointly solving segmentation, detection, and recognition for all major biomedical image modalities, paving the path for efficient and accurate image-based biomedical discovery.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2405.12971

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • BiomedParseData provides diverse, high-quality masks that supply rich pixel-level details for fine-grained analysis and serve as reliable supervision for synthesizing click-based annotation trajectories.

    “The dataset provides diverse, high-quality masks that supply rich pixel-level details for fine-grained analysis and serve as reliable supervision for synthesizing click-based annotation trajectories.”

    From IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation (2026) · cites this paper as [46]

    Details of BiomedParseData (p. 8) describes a diverse dataset of pixel-level segmentation masks and stringent criteria for including them, supporting the claim about detailed, high-quality masks. The paper does not describe using those masks to synthesize click-based annotation trajectories; the Overview (p. 3) instead describes text-based training. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • BiomedParseData contains 3.4 million image-mask-label tuples that cover 82 biomedical object types across 9 imaging modalities.

    “To this end, we utilize the large-scale BiomedParseData [46], containing 3.4 million image-mask-label tuples that cover 82 biomedical object types across 9 imaging modalities.”

    From IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation (2026) · cites this paper as [46]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 47

  1. [4][4] Salpea, N., Tzouveli, P. & Kollias, D. Medical image segmentation: A review of modern architectures. In European Conference on Computer Vision , 691-708 (Springer, 2022). -
  2. [7][7] Jiang, H. et al. A review of deep learning-based multiple-lesion recognition from medical images: classification, detection and segmentation. Computers in Biology and Medicine 157 , 106726 (2023). -
  3. [8][8] Kirillov, A. et al. Segment anything. arXiv preprint arXiv:2304.02643 (2023). - · arXiv
  4. [10][10] Tu, Z., Chen, X., Yuille, A. L. & Zhu, S.-C. Image parsing: Unifying segmentation, detection, and recognition. International Journal of computer vision 63 , 113-140 (2005). -
  5. [11][11] Tighe, J. & Lazebnik, S. Superparsing: scalable nonparametric image parsing with superpixels. International Journal of Computer Vision 101 , 329-349 (2013). -
  6. [12][12] Zhou, S. K. Medical image recognition, segmentation and parsing: machine learning and multiple object approaches (Academic Press, 2015). -
  7. [13][13] Gamper, J. et al. Pannuke dataset extension, insights and baselines. arXiv preprint arXiv:2003.10778 (2020). - · arXiv
  8. [14][14] Ji, Y . et al. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. arXiv preprint arXiv:2206.08023 (2022). - · arXiv
  9. [15][15] Bernard, O. et al. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: Is the problem solved? IEEE Transactions on Medical Imaging 37 , 2514-2525 (2018). -
  10. [17][17] Yang, J., Li, C., Dai, X. & Gao, J. Focal modulation networks. Advances in Neural Information Processing Systems 35 , 4203-4217 (2022). -
  11. [18][18] Gu, Y. et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH) 3 , 1-23 (2021). -
  12. [19][19] Sirinukunwattana, K., Snead, D. R. J. & Rajpoot, N. M. A stochastic polygons model for glandular structures in colon histology images. IEEE Transactions on Medical Imaging 34 , 2366-2378 (2015). -
  13. [20][20] Ronneberger, O., Fischer, P. & Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18 , 234-241 (Springer, 2015). -
  14. [21][21] C ¸ ic ¸ek, ¨ O., Abdulkadir, A., Lienkamp, S. S., Brox, T. & Ronneberger, O. 3D U-Net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and computer-assisted intervention , 424-432 (Springer, 2016). -
  15. [22][22] Milletari, F., Navab, N. & Ahmadi, S.-A. V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV) , 565-571 (IEEE, 2016). -
  16. [23][23] Li, X. et al. H-DenseUNet: hybrid densely connected unet for liver and tumor segmentation from CT volumes. IEEE transactions on medical imaging 37 , 2663-2674 (2018). -
  17. [24][24] Zhou, Z., Siddiquee, M. M. R., Tajbakhsh, N. & Liang, J. UNet++: Redesigning skip connections to exploit multiscale features in image segmentation. IEEE transactions on medical imaging 39 , 18561867 (2019). -
  18. [25][25] Myronenko, A. 3D MRI brain tumor segmentation using autoencoder regularization. In International MICCAI Brainlesion Workshop , 311-320 (Springer, 2018). -
  19. [27][27] Lee, H. H., Bao, S., Huo, Y. & Landman, B. A. 3D UX-Net: A large kernel volumetric convnet modernizing hierarchical transformer for medical image segmentation. arXiv preprint arXiv:2209.15076 (2022). - · arXiv
  20. [28][28] Lee, H. H. et al. Scaling up 3D kernels with bayesian frequency re-parameterization for medical image segmentation. arXiv preprint arXiv:2303.05785 (2023). - · arXiv
  21. [29][29] Lee, H. H. et al. DeformUX-Net: Exploring a 3D foundation backbone for medical image segmentation with depthwise deformable convolution. arXiv preprint arXiv:2310.00199 (2023). - · arXiv
  22. [30][30] Chen, J. et al. TransUNet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 (2021). - · arXiv
  23. [31][31] Xu, G., Zhang, X., He, X. & Wu, X. LeViT-UNet: Make faster encoders with transformer for medical image segmentation. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV) , 42-53 (Springer, 2023). -
  24. [32][32] Xie, Y., Zhang, J., Shen, C. & Xia, Y. Cotr: Efficiently bridging CNN and transformer for 3D medical image segmentation. In International conference on medical image computing and computer-assisted intervention , 171-180 (Springer, 2021). -
  25. [33][33] Wang, W. et al. TransBTS: Multimodal brain tumor segmentation using transformer. In International Conference on Medical Image Computing and Computer-Assisted Intervention , 109-119 (Springer, 2021). -
  26. [34][34] Hatamizadeh, A. et al. UNETR: Transformers for 3D medical image segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 574-584 (2022). -
  27. [35][35] Hatamizadeh, A. et al. Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images. In International MICCAI Brain lesion Workshop , 272-284 (Springer, 2022). -
  28. [36][36] Zhou, H.-Y. et al. nnFormer: Interleaved transformer for volumetric segmentation. arXiv preprint arXiv:2109.03201 (2021). - · arXiv
  29. [37][37] Cao, H. et al. Swin-UNet: UNet-like pure transformer for medical image segmentation. arXiv preprint arXiv:2105.05537 (2021). - · arXiv
  30. [40][40] Litjens, G. et al. A survey on deep learning in medical image analysis. Medical Image Analysis 42 , 60-88 (2017). URL http://dx.doi.org/10.1016/j.media.2017.07.005. - · DOI · Paper page
  31. [44][44] Girshick, R. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision , 1440-1448 (2015). -
  32. [45][45] He, K., Gkioxari, G., Doll´ ar, P. & Girshick, R. Mask R-CNN. In Proceedings of the IEEE international conference on computer vision , 2961-2969 (2017). -
  33. [46][46] Schmidt, U., Weigert, M., Broaddus, C. & Myers, G. Cell detection with star-convex polygons. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11 , 265-273 (Springer, 2018). -
  34. [48][48] Yang, H. et al. CircleNet: Anchor-free glomerulus detection with circle representation. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part IV 23 , 35-44 (Springer, 2020). -
  35. [49][49] Nguyen, E. H. et al. CircleSnake: Instance segmentation with circle representation. In International Workshop on Machine Learning in Medical Imaging , 298-306 (Springer, 2022). -
  36. [51][51] H¨ orst, F. et al. Cellvit: Vision transformers for precise cell segmentation and classification. arXiv preprint arXiv:2306.15350 (2023). - · arXiv
  37. [53][53] Gu, Y. et al. BiomedJourney: Counterfactual biomedical image generation by instruction-learning from multimodal patient journeys. arXiv preprint arXiv:2310.10765 (2023). - · arXiv
  38. [54][54] Li, C. et al. LLaVA-Med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36 (2024). -
  39. [55][55] Wong, H. E., Rakic, M., Guttag, J. & Dalca, A. V. Scribbleprompt: Fast and flexible interactive segmentation for any medical image. arXiv preprint arXiv:2312.07381 (2023). - · arXiv
  40. [56][56] Zou, X. et al. Segment everything everywhere all at once. Advances in Neural Information Processing Systems 36 (2024). -
  41. [57][57] Ren, T. et al. Grounded SAM: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401.14159 (2024). - · arXiv
  42. [58][58] Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F. & Adam, H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV) , 801-818 (2018). -
  43. [59][59] Lee, P., Goldberg, C. & Kohane, I. The AI revolution in medicine: GPT-4 and beyond (Pearson, 2023). -
  44. [60][60] Achiam, J. et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 (2023). - · arXiv
  45. [61][61] Massey Jr, F. J. The Kolmogorov-Smirnov test for goodness of fit. Journal of the American statistical Association 46 , 68-78 (1951). -
  46. [62][62] Canny, J. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine intelligence 679-698 (1986). -
  47. [63][63] Viola, P. & Jones, M. Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001 , vol. 1, I-I (Ieee, 2001). -

Cited by

Sorted by citations

Authors
Zhao, T., Gu, Y., Yang, J., Usuyama, N., Lee, H. H., Naumann, T., Gao, J., Crabtree, A., Abel, J., Moung-Wen, C., Piening, B., Bifulco, C., Wei, M., Poon, H., & Wang, S.
Affiliations
  • Microsoft (United States) (US)
  • Providence Portland Medical Center (US)
  • University of Washington (US)
Corresponding author
  • Wei, Mu
  • Poon, Hoifung
  • Wang, Sheng
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
Volume
22
Issue
1
Pages
166-176
ISSN
1548-7091 (ISSN-L)
DOI
10.48550/arxiv.2405.12971
arXiv
2405.12971
OpenAlex
W4404460707
Cited by
118
References
0
Citations by year
2026: 56 · 2025: 60 · 2024: 2
FWCI
16.40
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Nature Methods (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
medical image analysis, multi-modality imaging, natural language descriptions, image segmentation, object detection, object recognition, irregularly shaped objects
MeSH terms
Algorithms, Humans, Image Processing, Computer-Assisted, Pattern Recognition, Automated

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review