A medical multimodal large language model for future pandemics
Open accessPublished version of a preprint
Abstract
Deep neural networks have been integrated into the whole clinical decision procedure which can improve the efficiency of diagnosis and alleviate the heavy workload of physicians. Since most neural networks are supervised, their performance heavily depends on the volume and quality of available labels. However, few such labels exist for rare diseases (e.g., new pandemics). Here we report a medical multimodal large language model (Med-MLLM) for radiograph representation learning, which can learn broad medical knowledge (e.g., image understanding, text semantics, and clinical phenotypes) from unlabelled data. As a result, when encountering a rare disease, our Med-MLLM can be rapidly deployed and easily adapted to them with limited labels. Furthermore, our model supports medical data across visual modality (e.g., chest X-ray and CT) and textual modality (e.g., medical report and free-text clinical note); therefore, it can be used for clinical tasks that involve both visual and textual data. We demonstrate the effectiveness of our Med-MLLM by showing how it would perform using the COVID-19 pandemic “in replay”. In the retrospective setting, we test the model on the early COVID-19 datasets; and in the prospective setting, we test the model on the new variant COVID-19-Omicron. The experiments are conducted on 1) three kinds of input data; 2) three kinds of downstream tasks, including disease reporting, diagnosis, and prognosis; 3) five COVID-19 datasets; and 4) three different languages, including English, Chinese, and Spanish. All experiments show that our model can make accurate and robust COVID-19 decision-support with little labelled data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs can serve as classifiers in clinical prediction tasks and have achieved outstanding performance.
“When provided with clinical text inputs, LLMs can also serve as classifiers in clinical prediction tasks and have achieved outstanding performance31–33.”
From MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling (2025) · cites this paper as [32]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · see citing works
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · see citing works
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,745 · see citing works
Momentum Contrast for Unsupervised Visual Representation Learning
He, K., Fan, H., Wu, Y., Xie, S., & Girshick, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00975 · Cited by 13,347 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141 · see citing works
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · see citing works
Self-citationThe unified medical language system (UMLS): integrating biomedical terminology
Bodenreider, O. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh061 · Cited by 4,356 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.03771 · Cited by 3,347 · see citing works
Sensitivity of Chest CT for COVID-19: Comparison to RT-PCR
Fang, Y., Zhang, H., Xie, J., Lin, M., Ying, L., Pang, P., & Ji, W. (2020) · Radiology · doi:10.1148/radiol.2020200432 · Cited by 3,340 · see citing works
Physician burnout: contributors, consequences and solutions
West, C. P., Dyrbye, L. N., & Shanafelt, T. D. (2018) · Journal of Internal Medicine · doi:10.1111/joim.12752 · Cited by 2,768 · see citing works
Automated detection of COVID-19 cases using deep neural networks with X-ray images
Öztürk, T., Talo, M., Yildirim, E. A., Baloğlu, U. B., Yıldırım, Ö., & Acharya, U. R. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103792 · Cited by 2,607 · see citing works
Improved Baselines with Momentum Contrastive LearningPreprint
Chen, X., Fan, H., Girshick, R., & He, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04297 · Cited by 2,579 · see citing works
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · see citing works
Coronavirus Disease 2019 (COVID-19): A Perspective from China
Zu, Z. Y., Jiang, M. D., Xu, P. P., Chen, W., Ni, Q. Q., Lu, G. M., & Zhang, L. J. (2020) · Radiology · doi:10.1148/radiol.2020200490 · Cited by 2,070 · see citing works
Narin, A., Kaya, C., & Pamuk, Z. (2021) · Pattern Analysis and Applications · doi:10.1007/s10044-021-00984-y · Cited by 2,054 · see citing works
2012 IEEE Conference on Computer Vision and Pattern Recognition
Loy, C., Hospedales, T. M., Xiang, T., & Gong, S. (2012) · Cited by 1,883 · see citing works
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816 · see citing works
Alsentzer, E., Murphy, J., Boag, W., Weng, W.-H., Jindi, D., Naumann, T., & McDermott, M. (2019) · Proceedings of the 2nd Clinical Natural Language Processing Workshop · doi:10.18653/v1/w19-1909 · Cited by 1,736 · see citing works
Toward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734 · see citing works
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · see citing works
An overview of MetaMap: historical perspective and recent advances
Aronson, A. R., & Lang, F.-M. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2009.002733 · Cited by 1,519 · see citing works
Allocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties
Sinsky, C., Colligan, L., Li, L., Prgomet, M., Reynolds, S., Goeders, L., Westbrook, J., Tutty, M., & Blike, G. (2016) · Annals of Internal Medicine · doi:10.7326/m16-0961 · Cited by 1,494 · cited 1× · see citing works
“Given the large number of COVID-19 patients, writing medical reports is a heavy burden for physicians who could otherwise concentrate on patient care 18,19 .”
Transformers in medical imaging: A survey
Shamshad, F., Khan, S., Zamir, S. W., Khan, M. H., Hayat, M., Khan, F. S., & Fu, H. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102802 · Cited by 1,412 · see citing works
Zhou, S. K., Greenspan, H. K., Davatzikos, C. A., Duncan, J. S., van Ginneken, B., Madabhushi, A., Prince, J. L., Rueckert, D., & Summers, R. M. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3054390 · Cited by 1,292 · see citing works
The Role of Chest Imaging in Patient Management During the COVID-19 Pandemic
Rubin, G. D., Ryerson, C. J., Haramati, L. B., Sverzellati, N., Kanne, J. P., Raoof, S., Schluger, N. W., Volpi, A., Yim, J., Martin, I. B. K., et al. (2020) · CHEST Journal · doi:10.1016/j.chest.2020.04.003 · Cited by 1,249 · see citing works
Exploring the effect of image enhancement techniques on COVID-19 detection using chest X-ray images
Rahman, T., Khandakar, A. A., Qiblawey, Y., Tahir, A., Serkan Kiranyaz, B., Kashem, S. B. A., Islam, M. T., Maadeed, S. A., Zughaier, S. M., Khan, M. S., et al. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104319 · Cited by 1,153 · see citing works
Imaging Profile of the COVID-19 Infection: Radiologic Findings and Literature Review
Ng, M., Lee, E. Y. P., Jin, Y., Yang, F., Li, X., Wang, H., Lui, M. M., Lo, C. S.-Y., Leung, B., Khong, P., et al. (2020) · Radiology Cardiothoracic Imaging · doi:10.1148/ryct.2020200034 · Cited by 1,141 · see citing works
Rubin, G. D., Ryerson, C. J., Haramati, L. B., Sverzellati, N., Kanne, J. P., Raoof, S., Schluger, N. W., Volpi, A., Yim, J.-J., Martin, I. B. K., et al. (2020) · Radiology · doi:10.1148/radiol.2020201365 · Cited by 1,101 · see citing works
Roberts, M., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Aviles-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00307-0 · Cited by 1,044 · cited 4× · see citing works
“…this manner, Med-MLLM handles visual, textual and multi-modal input, and therefore can be applied to COVID-19 reporting (i.e., medical report generation), diagnosis (i.e., disease classi fi cation), and prognosis (i.e., survival prediction) tasks with limited labels for training 1,2,12,13,15,27,35 .”
InfluentialCOVID-19 Image Data Collection: Prospective Predictions are the Future
Cohen, J., Morrison, P., Dao, L., Roth, K., Duong, T. Q., & Ghassemi, M. (2020) · The Journal of Machine Learning for Biomedical Imaging · doi:10.59275/j.melba.2020-48g7 · Cited by 1,026 · see citing works
ChatGPT and other large language models are double-edged swords
Shen, Y., Heacock, L., Elias, J., Hentel, K. D., Reig, B., Shih, G., & Moy, L. (2023) · Radiology · doi:10.1148/radiol.230163 · Cited by 1,008 · see citing works
Auto-encoder based dimensionality reduction
Wang, Y., Yao, H., & Zhao, S. (2015) · Neurocomputing · doi:10.1016/j.neucom.2015.08.104 · Cited by 983 · see citing works
Two public chest X-ray datasets for computer-aided screening of pulmonary diseases.
(2014) · PubMed Central · doi:10.3978/j.issn.2223-4292.2014.11.20 · Cited by 918 · see citing works
Roberts, M. T., Driggs, D., Thorpe, M., Gilbey, J., Yeung, M., Ursprung, S., Avilés-Rivero, A. I., Etmann, C., McCague, C., Beer, L., et al. (2020) · Apollo (University of Cambridge) · doi:10.17863/cam.64000 · Cited by 902 · see citing works
Peng, Y., Yan, S., & Lu, Z. (2019) · Proceedings of the 18th BioNLP Workshop and Shared Task · doi:10.18653/v1/w19-5006 · Cited by 899 · see citing works
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847 · see citing works
MedCLIP: Contrastive Learning from Unpaired Medical Images and Text
Wang, Z., Wu, Z., Agarwal, D., & Sun, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.256 · Cited by 838 · see citing works
COVID-19 Image Data CollectionPreprint
Cohen, J., Morrison, P., & Dao, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.11597 · Cited by 722 · see citing works
Tawfik, D. S., Profit, J., Morgenthaler, T. I., Satele, D., Sinsky, C. A., Dyrbye, L. N., Tutty, M., West, C. P., & Shanafelt, T. D. (2018) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2018.05.014 · Cited by 716 · see citing works
Generating Radiology Reports via Memory-driven Transformer
Chen, Z., Song, Y., Chang, T.-H., & Wan, X. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.112 · Cited by 693 · see citing works
On the Automatic Generation of Medical Imaging Reports
Jing, B., Xie, P., & Xing, E. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1240 · Cited by 657 · cited 4× · see citing works
“…(EMR), Transformer based frameworks 5 have been be applied to estimate the mortality or re-hospitalisation probabilities 6 ; For multi-modal data such as radiology image-report pairs, the encoder-decoder based frameworks 7 – 11 have been applied to generate textual reports from medical images.”
InfluentialSelf-supervised learning for medical image analysis using image context restoration
Chen, L., Bentley, P., Mori, K., Misawa, K., Fujiwara, M., & Rueckert, D. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101539 · Cited by 575 · see citing works
Wang, X., Peng, Y., Lü, L., Lu, Z., & Summers, R. M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00943 · Cited by 574 · see citing works
Huang, S.-C., Shen, L., Lungren, M. P., & Yeung, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00391 · Cited by 489 · see citing works
Deep learning and medical image processing for coronavirus (COVID-19) pandemic: A survey
Bhattacharya, S., Maddikunta, P. K. R., Pham, Q., Gadekallu, T. R., Siva Rama Krishnan, S., Chowdhary, C. L., Alazab, M., & Piran, M. J. (2020) · Sustainable Cities and Society · doi:10.1016/j.scs.2020.102589 · Cited by 488 · see citing works
Tiu, E., Talius, E., Patel, P., Langlotz, C. P., Ng, A. Y., & Rajpurkar, P. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00936-9 · Cited by 442 · see citing works
Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation
Liu, F., Wu, X., Ge, S., Fan, W., & Zou, Y. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01354 · Cited by 396 · see citing works
Self-citationCheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.3301590 · Cited by 392 · see citing works
Recent Advances in Autoencoder-Based Representation LearningPreprint
Tschannen, M., Bachem, O., & Lučić, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.05069 · Cited by 359 · see citing works
Zhou, Z., Sodha, V., Pang, J., Gotway, M. B., & Liang, J. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101840 · Cited by 353 · see citing works
Shih, G., Wu, C. C., Halabi, S. S., Kohli, M. D., Prevedello, L. M. S., Cook, T. S., Sharma, A., Amorosa, J. K., Arteaga, V. A., Galperin-Aizenberg, M., et al. (2019) · Radiology Artificial Intelligence · doi:10.1148/ryai.2019180041 · Cited by 341 · see citing works
Towards Expert-Level Medical Question Answering with Large Language Models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Hou, L., Clark, K., Pfohl, S., Cole-Lewis, H., Neal, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.09617 · Cited by 338 · see citing works
Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing
Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., et al. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20059-5_1 · Cited by 317 · cited 15× · see citing works
“In particular, when transferring our approach trained on patient data from Spain to China, we observe an encouraging performance, i.e., 90.1% AUC, which is competitive with the region-speci fi c results of previous works CLIP (80.7% AUC) and BioViL (90.4% AUC), which were obtained by training and testing on the data co…”
InfluentialChatGPT is shaping the future of medical writing but still requires human judgment
Kitamura, F. C. (2023) · Radiology · doi:10.1148/radiol.230171 · Cited by 317 · see citing works
BIMCV COVID-19+: a large annotated dataset of RX and CT images from COVID-19 patientsPreprint
MdlI, V., Saborit-Torres, J. M., Ja, M., Pertusa, A., Bustos, A., Cazorla, M., Galant, J., Barber, X., Orozco‐Beltrán, D., García-García, F., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.01174 · Cited by 246 · see citing works
A systematic review of natural language processing applied to radiology reports
Casey, A., Davidson, E., Poon, M., Dong, H., Duma, D., Grivas, A., Grover, C., Suárez-Paniagua, V., Tobin, R., Whiteley, W., et al. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01533-7 · Cited by 234 · see citing works
Fountain, T., & Lapata, M. (2012) · The Association for Computational Linguistics · Cited by 217 · see citing works
NegBio: a high-performance tool for negation and uncertainty detection in radiology reports
Peng, Y., Wang, X., Lu, L., Bagheri, M., Summers, R., & Lu, Z. (2017) · arXiv · doi:10.48550/arxiv.1712.05898 · Cited by 175 · see citing works
Contrastive Attention for Automatic Chest X-ray Report Generation
Liu, F., Yin, C., Wu, X., Ge, S., Zhang, P., & Sun, X. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.23 · Cited by 175 · see citing works
Self-citationYou, D., Liu, F., Ge, S., Xie, X., Zhang, J., & Wu, X. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87199-4_7 · Cited by 170 · see citing works
Self-citationA systematic review of natural language processing applied to radiology reports.
Casey, A., Davidson, E. M., Poon, M. T. C., Dong, H., Duma, D., Grivas, A., Grover, C., Suárez-Paniagua, V., Tobin, R., Whiteley, W. N., et al. (2021) · UCL Discovery (University College London) · Cited by 160 · see citing works
Competence-based Multimodal Curriculum Learning for Medical Report Generation
Liu, F., Ge, S., & Wu, X. (2021) · doi:10.18653/v1/2021.acl-long.234 · Cited by 145 · see citing works
Self-citationHaghighi, F., Taher, M. R. H., Zhou, Z., Gotway, M. B., & Liang, J. (2021) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2021.3060634 · Cited by 144 · see citing works
Zhou, H.-Y., Chen, X., Zhang, Y., Luo, R., Wang, L., & Yu, Y. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-021-00425-9 · Cited by 126 · see citing works
Cross-Modal Prototype Driven Network for Radiology Report Generation
Wang, J., Bhalerao, A. H., & He, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_33 · Cited by 108 · see citing works
Zhou, H.-Y., Yu, S., Bian, C., Hu, Y., Ma, K., & Zheng, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59710-8_39 · Cited by 100 · see citing works
Self-citationSoomro, T. A., Zheng, L., Afifi, A. J., Ali, A., Yin, M., & Gao, J. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-09985-z · Cited by 96 · see citing works
The Influence of Information Technology on Patient-Physician Relationships
Weiner, M., & Biondich, P. (2006) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2006.00307.x · Cited by 91 · see citing works
Peng, Y., Tang, Y., Lee, S., Zhu, Y., Summers, R. M., & Lu, Z. (2020) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2020.3035935 · Cited by 90 · see citing works
Facing the New Covid-19 Reality
El-Sadr, W. M., Vasan, A., & El-Mohandes, A. A. E. (2023) · New England Journal of Medicine · doi:10.1056/nejmp2213920 · Cited by 84 · see citing works
Auto-Encoding Knowledge Graph for Unsupervised Medical Report GenerationPreprint
Liu, F., You, C., Wu, X., Ge, S., Wang, S., & Sun, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.04318 · Cited by 80 · see citing works
Self-citationLiu, G., Liao, Y., Wang, F., Zhang, B., Zhang, L., Liang, X., Wan, X., Li, S., Li, Z., Zhang, S., et al. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3099165 · Cited by 79 · see citing works
Deep learning and its role in COVID-19 medical imaging
Desai, S. B., Pareek, A., & Lungren, M. P. (2020) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2020.100013 · Cited by 77 · see citing works
Vu, Y. N. T., Wang, R., Balachandar, N., Liu, C., Ng, A. Y., & Rajpurkar, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.10663 · Cited by 63 · see citing works
Pavlova, M., Terhljan, N., Chung, A. G., Zhao, A., Surana, S., Aboutalebi, H., Gunraj, H., Sabri, A., Alaref, A. A., & Wong, A. M. (2022) · Frontiers in Medicine · doi:10.3389/fmed.2022.861680 · Cited by 49 · see citing works
Lyu, W., Dong, X., Wong, R., Zheng, S., Abell-Hart, K., Wang, F., & Chen, C. (2022) · arXiv · doi:10.48550/arxiv.2208.10240 · Cited by 46 · cited 1× · see citing works
“…Electronic Medical Record (EMR), Transformer based frameworks 5 have been be applied to estimate the mortality or re-hospitalisation probabilities 6 ; For multi-modal data such as radiology image-report pairs, the encoder-decoder based frameworks 7 – 11 have been applied to generate textual…”
A Novel Deep Convolutional Neural Network Model for COVID-19 Disease Detection
Irmak, E. (2020) · Medical Technologies Congress (TIPTEKNO) · doi:10.1109/tiptekno50054.2020.9299286 · Cited by 41 · see citing works
Finding COVID-19 from Chest X-rays using Deep Learning on a Small DatasetPreprint
Hall, L., Goldgof, D. B., Paul, R., et al. (2020) · doi:10.36227/techrxiv.12083964.v1 · Cited by 40 · see citing works
Carlile, M., Hurt, B., Hsiao, A., Hogarth, M. A., Longhurst, C., & Dameff, C. J. (2020) · Journal of the American College of Emergency Physicians Open · doi:10.1002/emp2.12297 · Cited by 37 · see citing works
Driggs, D., SELBY, I. A., Roberts, M. T., Gkrania‐Klotsas, E., Rudd, J. H. F., Yang, G., Babar, J. L., Sala, E., & Schönlieb, C. (2021) · Radiology Artificial Intelligence · doi:10.1148/ryai.2021210011 · Cited by 30 · see citing works
DeltaNet:Conditional Medical Report Generation for COVID-19 DiagnosisPreprint
Wu, X. W., Yang, S., Qiu, Z., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.13229 · Cited by 18 · see citing works
Self-citationZhou, H.-Y., Yu, S., Bian, C., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.07423 · Cited by 11 · see citing works
Finding Covid-19 from Chest X-rays using Deep Learning on a Small DatasetPreprint
Hall, L., Paul, R., Goldgof, D. B., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02060 · Cited by 7 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 15
- [3]Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (Curran Associates, Inc., 2012).
- [5]Vaswani, A. et al. Attention is all you need. In Advances in Neural Information Processing Systems (Curran Associates, Inc., 2017).
- [8]Liu, G. et al. Clinically accurate chest x-ray report generation. In Machine Learning for Healthcare Conference (PMLR, 2019).
- [9]Li, Y., Liang, X., Hu, Z. & Xing, E. P. Hybrid retrieval-generation reinforced agent for medical image report generation. In Advances in Neural Information Processing Systems (Curran Associates, Inc., 2018).
- [23]Liu, F. et al. Retrieve, reason, and refine: Generating accurate and faithful patient instructions. In Advances in Neural Information Processing Systems (Curran Associates, Inc., 2022).
- [28]Brown, T. et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (Curran Associates, Inc., 2020).
- [31]Zhou, H.-Y., Lian, C., Wang, L. & Yu, Y. Advancing radiograph representation learning with masked record modeling. In The Eleventh International Conference on Learning Representations (OpenReview.net, 2023).
- [34]Zhang, Y., Jiang, H., Miura, Y., Manning, C. D. & Langlotz, C. P. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, 2–25 (2022).
- [46]Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. E. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (PMLR, 2020).
- [47]National Institutes of Health. PubMed Corpora, https://pubmed.ncbi.nlm.nih.gov/download/ (2022).
- [66]Lin, C.-Y. ROUGE: A package for automatic evaluation of summaries. In Annual Meeting of the Association for Computational Linguistics (Association for Computational Linguistics, 2004).
- [87]Kiyasseh, D., Zhu, T. & Clifton, D. A. Clocs: Contrastive learning of cardiac signals across space, time, and patients. In International Conference on Machine Learning, 5606–5615 (PMLR, 2021).
- [97]Preechakul, K. et al. Set prediction in the latent space. In Advances in Neural Information Processing Systems, 25516–25527 (Curran Associates, Inc., 2021).
- [98]Liu, F., Liu, Y., Ren, X., He, X. & Sun, X. Aligning visual regions and textual concepts for semantic-grounded image representations. In Adv. Neural Inf. Proces. Syst. (Curran Associates, Inc., 2019).
- [103]Loshchilov, I. & Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations (OpenReview.net, 2019).
Cited by
Sorted by citations
Quantifying the uncertainty of LLM hallucination spreading in complex adaptive social networks
Hao, G., Wu, J., Pan, Q., & Morello, R. (2024) · Scientific Reports · doi:10.1038/s41598-024-66708-4 · Cited by 36 · see citing works
Toward Automated Simulation Research Workflow through LLM Prompt Engineering Design
Liu, Z., Chai, Y., & Li, J. (2025) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c01653 · Cited by 24 · see citing works
Self-citationDu, X., Zhou, Z., Wang, Y., Chuang, Y.-W., Li, Y., Yang, R., Hong, P., Bates, D. W., & Zhou, L. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2025.106091 · Cited by 26 · see citing works
Large language model for medical images: a survey of taxonomy, systematic review, and future trends
Wang, P., Lu, W., Lu, C., Zhou, R., Li, M., & Qin, L. (2025) · Big Data Mining and Analytics · doi:10.26599/bdma.2024.9020090 · Cited by 21 · see citing works
MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling
Gao, J., Rahman, M., Caskey, J., Oguss, M., O’Rourke, A., Brown, R., Stey, A., Mayampurath, A., Churpek, M. M., Chen, G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02219-4 · Cited by 2 · see citing works
“Published in partnership with Seoul National University Bundang Hospital https://doi.org/10.1038/s41746-025-02219-4 MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling”
InfluentialArtificial intelligence language models for medical text analysis: A systematic review
Sorayaie Azar, A., Bagherzadeh Mohasefi, J., Wiil, U. K., Naemi, A., & Ebrahimi, A. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103441 · Cited by 2 · see citing works
- Authors
- Liu, F., Zhu, T., Wu, X., Yang, B., You, C., Wang, C., Lu, L., Liu, Z., Zheng, Y., Sun, X., Yang, Y., Clifton, L., & Clifton, D. A.
- Affiliations
- University of Oxford (GB)
- Tencent (China) (CN)
- Peking University (CN)
- Yale University (US)
- Shanghai Jiao Tong University (CN)
- Corresponding author
- Liu, Fenglin
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2023-12-02
- Volume
- 6
- Issue
- 1
- Pages
- 226-226
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC10693607
- OpenAlex
- W4389267014
- Cited by
- 155
- References
- 95
- Citations by year
- 2026: 40 · 2025: 77 · 2024: 38
- FWCI
- 14.01
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Portal (King's College London) (publishedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Greater South Information System
- Funders
- Royal Academy of Engineering
- DH | National Institute for Health Research
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- multimodal large language models, rare disease diagnosis, COVID-19 pandemic, Omicron variant, chest radiography, computed tomography, clinical decision support
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
