The DOI matches its Crossref record; 27 of the 28 references it cites were found in a registry or an index.
On the Automatic Generation of Medical Imaging Reports
Open accessAbstract
Medical imaging is widely used in clinical practice for diagnosis and treatment.Report-writing can be error-prone for unexperienced physicians, and timeconsuming and tedious for experienced physicians.To address these issues, we study the automatic generation of medical imaging reports.This task presents several challenges.First, a complete report contains multiple heterogeneous forms of information, including findings and tags.Second, abnormal regions in medical images are difficult to identify.Third, the reports are typically long, containing multiple sentences.To cope with these challenges, we (1) build a multi-task learning framework which jointly performs the prediction of tags and the generation of paragraphs, (2) propose a co-attention mechanism to localize regions containing abnormalities and generate narrations for them, (3) develop a hierarchical LSTM model to generate long paragraphs.We demonstrate the effectiveness of the proposed methods on two publicly available datasets.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Recently, deep learning methods have also been used for automated generation of radiology reports , typically incorporating long-short-term-memory (LSTM) network models to generate the textual paragraphs [314, 315, 316, 317], and also to identify findings in radiology reports [318, 319, 320].”
From An overview of deep learning in medical imaging focusing on MRI (2019) · cites this paper as [314]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 5×
“Results in Table 3 show that Ours-CoAttention and VGG-19 [17] network performs very similar for tag prediction.”
BackgroundMethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“than adopting a single-layer LSTM (Hochreiter and Schmidhuber, 1997), which is less capable of modeling long word sequences, we leverage the compositional nature of the report and adopt a hierarchical LSTM to produce long texts.”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“We used the Adam (Kingma and Ba, 2014) optimizer for parameter learning.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 7×
“Note that the only difference between Ours-NoAttention and CNN-RNN [18] in Table 1 is that Ours-NoAttention adopts a hierarchical LSTM decoder while CNNRNN [18] only adopts a single-layer LSTM.”
BackgroundMethodInfluentialLong-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · cited 2×
“We report the paragraph generation (upper part of Table 1) and one sentence generation (lower part of Table 1) results using the following captioning evaluation tools: BLEU [13], METEOR [4], ROUGE [12] and CIDER [17].”
MethodMeteor Universal: Language Specific Translation Evaluation for Any Target Language
Denkowski, M., & Lavie, A. (2014) · WMT@ACL · doi:10.3115/v1/w14-3348 · Cited by 2,108 · cited 2×
“We report the paragraph generation (upper part of Table 1) and one sentence generation (lower part of Table 1) results using the following captioning evaluation tools: BLEU [13], METEOR [4], ROUGE [12] and CIDER [17].”
MethodImage Captioning with Semantic Attention
You, Q., Jin, H., Wang, Z., Chen, F., & Luo, J. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.503 · Cited by 1,869 · cited 6×
“We first compare our method with several state-of-the-art image captioning models: CNN-RNN [19], LRCN [5], Soft ATT [20], ATT-RK [21].”
BackgroundMethodInfluentialFrom captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298754 · Cited by 1,321
DenseCap: Fully Convolutional Localization Networks for Dense Captioning
Johnson, J., Karpathy, A., & Fei-Fei, L. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.494 · Cited by 1,199 · cited 2×
“Most of recent image captioning models are based on a CNN-RNN framework [18, 6, 9, 19, 20, 8, 11].”
BackgroundMethodPreparing a collection of radiology examinations for distribution and retrieval
Demner-Fushman, D., Kohli, M. D., Rosenman, M. B., Shooshan, S. E., Rodriguez, L., Antani, S., Thoma, G. R., & McDonald, C. J. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv080 · Cited by 1,164
Deep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048
Kastner, S., & Buschman, T. J. (2014) · doi:10.1093/acrefore/9780190264086.013.79 · Cited by 820 · cited 1×
“Previous works have shown that visual attention alone can perform fairly well for localizing the objects [1] and aiding caption generation [19].”
TagProp: Discriminative metric learning in nearest neighbor models for image auto-annotation
Guillaumin, M., Mensink, T., Verbeek, J. J., & Schmid, C. (2009) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2009.5459266 · Cited by 678 · cited 1×
“As for the training loss of the multi-label classification (MLC) task, since the number of tags for semantic attention is fixed as 10, we treat MLC as a multilabel retrieval task and adopt a softmax crossentropy loss (a multi-label ranking loss), similar to (Gong et al., 2013; Guillaumin et al., 2009).”
MethodHow to Construct Deep Recurrent Neural NetworksPreprint
Pascanu, R., Gülçehre, Ç., Cho, K., & Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.6026 · Cited by 574 · cited 1×
“Topic Generator We use a deep output layer [14] to strengthen the context information in topic vector t.”
MethodLearning to Read Chest X-Rays: Recurrent Neural Cascade Model for Automated Image Annotation
Shin, H.-C., Roberts, K. E., Lü, L., Demner‐Fushman, D., Yao, J., & Summers, R. M. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.274 · Cited by 401
A Hierarchical Approach for Generating Descriptive Image Paragraphs
Krause, J., Johnson, J., Krishna, R., & Fei-Fei, L. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.356 · Cited by 377 · cited 4×
“Following [11], when generating words, topic vector t and the special START token are used as the first and the second input of the word LSTM, and the subsequent inputs are the word sequence.”
BackgroundMethodInfluentialMDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network
Zhang, Z., Xie, Y., Xing, F., McGough, M., & Yang, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.378 · Cited by 354
Deep Convolutional Ranking for Multilabel Image AnnotationPreprint
Gong, Y., Jia, Y., Leung, T., Toshev, A., & Ioffe, S. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.4894 · Cited by 327 · cited 1×
“Note that the loss function of both models are softmax cross-entropy loss [7].”
BackgroundRecurrent Topic-Transition GAN for Visual Paragraph Generation
Liang, X., Hu, Z., Zhang, H., Gan, C., & Xing, E. P. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.364 · Cited by 209
Self-citationMulti-Timescale Long Short-Term Memory Neural Network for Modelling Sentences and Documents
Liu, P., Qiu, X., Chen, X., Wu, S., & Huang, X. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1280 · Cited by 184 · cited 1×
“This result is not surprising since it is well-known that a single-layer LSTM cannot effectively model long sequences (Liu et al., 2015; Martin and Cundy, 2018).”
BackgroundParallelizing Linear Recurrent Neural Nets Over Sequence LengthPreprint
Martin, E., & Cundy, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.04057 · Cited by 77 · cited 1×
“irectly demonstrates the effectiveness of the hierarchical LSTM. This result is not surprising since it is well-known that a single-layer LSTM cannot effectively model long sequences (Liu et al.,2015;Martin and Cundy, 2018). Additionally, employing semantic attention alone (Ours-Semantic-only) or visual attention alon…”
BackgroundDifficulties in the Interpretation of Chest Radiography
Delrue, L., Gosselin, R., Ilsen, B., et al. (2011) · Medical radiology · doi:10.1007/978-3-540-79942-9_2 · Cited by 76
From medical image to automatic medical report generation
Kisilev, P., Walach, E., Barkan, E., et al. (2015) · IBM Journal of Research and Development · doi:10.1147/jrd.2015.2393193 · Cited by 52
BMJ Publishing Group Ltd, & Royal College of Psychiatrists and British Psychological Society (2008) · Evidence-Based Mental Health · doi:10.1136/ebmh.11.4.102 · Cited by 3 · cited 2×
“Most of recent image captioning models are based on a CNN-RNN framework [18, 6, 9, 19, 20, 8, 11].”
Method
Research works printed but not linked: 2
- Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. 2015. Multiple object recognition with visual atten- tion. ICLR.
- Chin-Yew Lin. 2004. Rouge: A package for auto- matic evaluation of summaries. In Text summariza- tion branches out: Proceedings of the ACL-04 work- shop, volume 8. Barcelona, Spain.
Cited by
Sorted by citations
An overview of deep learning in medical imaging focusing on MRI
Lundervold, A. S., & Lundervold, A. (2019) · Zeitschrift für Medizinische Physik · doi:10.1016/j.zemedi.2018.11.002 · Cited by 2,058
Generative adversarial network in medical imaging: A review
Xin, Y., Walia, E., & Babyn, P. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.101552 · Cited by 1,959
“Tang et al. (2018) SGAN L1, 2, 8 – M32 [ √ ] Denoising, contrast enhance Shan et al.”
Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
van der Velden, B. H. M., Kuijf, H. J., Gilhuijs, K. G. A., & Viergever, M. A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102470 · Cited by 1,328
Zhou, S. K., Greenspan, H. K., Davatzikos, C. A., Duncan, J. S., van Ginneken, B., Madabhushi, A., Prince, J. L., Rueckert, D., & Summers, R. M. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3054390 · Cited by 1,373
“6) Others technologies include landmark detection [18], image or view recognition [19], automatic report generation [20], and so on.”
BackgroundA General Survey on Attention Mechanisms in Deep Learning
Brauwers, G., & Frăsincar, F. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3126456 · Cited by 722
Metadata
- Authors
- Jing, B., Xie, P., & Xing, E.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2018
- Pages
- 2577-2586
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2770165365
- Cited by
- 657
- References
- 28
- Citations by year
- 2026: 57 · 2025: 123 · 2024: 115 · 2023: 111 · 2022: 80 · 2021: 81 · 2020: 45 · 2019: 34 · 2018: 11
- FWCI
- 12.69
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- multi-task learning, co-attention, hierarchical LSTM, tag prediction, clinical report writing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
