Learning to detect chest radiographs containing pulmonary lesions using visual attention networks
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Machine learning approaches hold great potential for the automated detection of lung nodules on chest radiographs, but training algorithms requires very large amounts of manually annotated radiographs, which are difficult to obtain. The increasing availability of PACS (Picture Archiving and Communication System), is laying the technological foundations needed to make available large volumes of clinical data and images from hospital archives. Binary labels indicating whether a radiograph contains a pulmonary lesion can be extracted at scale, using natural language processing algorithms. In this study, we propose two novel neural networks for the detection of chest radiographs containing pulmonary lesions. Both architectures make use of a large number of weakly-labelled images combined with a smaller number of manually annotated x-rays. The annotated lesions are used during training to deliver a type of visual attention feedback informing the networks about their lesion localisation performance. The first architecture extracts saliency maps from high-level convolutional layers and compares the inferred position of a lesion against the true position when this information is available; a localisation error is then back-propagated along with the softmax classification error. The second approach consists of a recurrent attention model that learns to observe a short sequence of smaller image portions through reinforcement learning; the reward function penalises the exploration of areas, within an image, that are unlikely to contain nodules. Using a repository of over 430,000 historical chest radiographs, we present and discuss the proposed methods over related architectures that use either weakly-labelled or annotated images only.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Stacked Hourglass Networks for Human Pose Estimation
Newell, A., Yang, K., & Deng, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_29 · Cited by 5,634 · cited 1×
“In [46], a stacked hourglass network is proposed to introduce bottom-up, top-down inference across multiple scales.”
BackgroundADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Backpropagation through time: what it does and how to do it
Werbos, P. J. (1990) · Proceedings of the IEEE · doi:10.1109/5.58337 · Cited by 4,879
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616
OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & LeCun, Y. (2013) · arXiv · doi:10.48550/arxiv.1312.6229 · Cited by 2,680 · cited 4×
“Nodule vs Normal Only Nodule vs All Others Method Accuracy F1 Sensitivity Precision Accuracy F1 Sensitivity Precision OverFeat [12] 0.”
BackgroundMethodInfluentialScalable Object Detection Using Deep Neural Networks
Erhan, D., Szegedy, C., Toshev, A., & Anguelov, D. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.276 · Cited by 1,161 · cited 1×
“The large majority of these methods use regression models to predict the coordinates of the bounding boxes [9,10] or, alternatively, make use of sliding windows [11,12].”
MethodSimple statistical gradient-following algorithms for connectionist reinforcement learning
Williams, R. J. (1992) · Machine Learning · doi:10.1023/a:1022672621406 · Cited by 817
Multiple Object Recognition with Visual Attention
Ba, J., Mnih, V., & Kavukcuoglu, K. (2014) · arXiv · doi:10.48550/arxiv.1412.7755 · Cited by 808
Human Pose Estimation with Iterative Error Feedback
Carreira, J., Agrawal, P., Fragkiadaki, K., & Malik, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.512 · Cited by 805 · cited 1×
“Feedback neural networks without recurrent connections have been used recently for human pose estimation [45], where a self-correcting model progressively changes the initial prediction by iteratively feeding back the error predictions.”
BackgroundDiagnostic Concordance Among Pathologists Interpreting Breast Biopsy Specimens
Elmore, J. G., Longton, G. M., Carney, P. A., Geller, B. M., Onega, T., Tosteson, A. N. A., Nelson, H. D., Pepe, M. S., Allison, K. H., Schnitt, S. J., et al. (2015) · JAMA · doi:10.1001/jama.2015.1405 · Cited by 751
End-to-End People Detection in Crowded Scenes
Stewart, R., Andriluka, M., & Ng, A. Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.255 · Cited by 539 · cited 2×
“The second algorithm uses a CNN module to encode an image in high-level feature representation, which is then passed to a LSTM (long-short term memory) network which learns to decode this representation into predicted bounding boxes ( Stewart and Andriluka, 2015 ).”
MethodDeep Learning is Robust to Massive Label Noise
Rolnick, D., Veit, A., Belongie, S., & Shavit, N. (2017) · arXiv · doi:10.48550/arxiv.1705.10694 · Cited by 402
MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network
Zhang, Z., Xie, Y., Xing, F., McGough, M., & Yang, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.378 · Cited by 354 · cited 2×
“Soft attention has been used for learning a mapping between radiological reports and the corresponding histopathology specimens (Zhang et al., 2017).”
BackgroundCho, J., Lee, K., Shin, E., Choy, G., & Do, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06348 · Cited by 268
Interleaved text/image Deep Mining on a large-scale radiology database
Shin, H.-C., Lu, L., Kim, L., Seff, A., Yao, J., & Summers, R. M. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298712 · Cited by 94
Research works printed but not linked: 31
- American Cancer Society, 1999. Key statistics for lung cancer. https://www.cancer.org/cancer/non-small-cell-lung-cancer/about/key-statistics.html, arXiv:1711.00583. · arXiv
- Anderson, 1994, The efficacy of double reading mammograms in breast screening, Clin. Radiol., 49, 248, 10.1016/S0009-9260(05)81850-1 · DOI
- Brady, 2017, Error and discrepancy in radiology: inevitable or avoidable?, Insights Imag., 8, 171, 10.1007/s13244-016-0534-1 · DOI · Paper page
- Chapman, 2013, Extending the negex lexicon for multiple languages, Stud. Health Technol. Inform., 677
- Cichy, 2014, Resolving human object recognition in space and time, Nat. Neurosci., 17, 455, 10.1038/nn.3635 · DOI · Paper page
- Cornia, 2016, A deep multi-level network for saliency prediction.
- Denil, 2011, Learning where to attend with deep architectures for image tracking.
- Elmore, 1994, Variability in radiologists’ interpretations of mammograms, N. Engl. J. Med., 331, 1493, 10.1056/NEJM199412013312206 · DOI · Paper page
- Everingham, 2010, The pascal visual object classes (VOC) challenge, Int. J. Comput. Vis., 88, 303, 10.1007/s11263-009-0275-4 · DOI · Paper page
- Ferlay, 2013, Cancer incidence and mortality patterns in europe: estimates for 40 countries in 2012, Eur. J. Cancer, 49, 1374, 10.1016/j.ejca.2012.12.027 · DOI · Paper page
- Forrest, 1981, Radiologic errors in patients with lung cancer, Western J. Med., 134, 485
- Guan, 2017, Who said what: modeling individual labelers improves classification.
- Hansell, 2008, Fleischner society: glossary of terms for thoracic imaging, Radiology, 246, 697, 10.1148/radiol.2462070712 · DOI · Paper page
- Langlotz, 2006, Radlex: a new method for indexing online educational materials, Radiographics, 26, 1595, 10.1148/rg.266065168 · DOI · Paper page
- Langlotz, 2006, Radlex: a new method for indexing online educational materials, Radiographics, 26, 1595, 10.1148/rg.266065168 · DOI · Paper page
- Lin, 2014, Microsoft coco: common objects in context.
- Manning, 2014, The stanford coreNLP natural language processing toolkit, In Association for Computational Linguistics (ACL) System Demonstrations, 55, 10.3115/v1/P14-5010 · DOI · Paper page
- Marbach, 2003, Approximate gradient methods in policy-space optimization of markov reward processes, Discret. Event Dyn. Syst., 13, 111, 10.1023/A:1022145020786 · DOI
- Mehrabi, 2015, DEEPEN: a negation detection system for clinical text incorporating dependency relation into negex, J. Biomed. Inform., 54, 213, 10.1016/j.jbi.2015.02.010 · DOI · Paper page
- Moore, 2011, Sensitivity and specificity of a cad solution for lung nodule detection on chest radiograph with cta correlation, J. Digit. Imag., 24, 405, 10.1007/s10278-010-9284-7 · doi:10.1007/s10278-010-9284-7
- Northcutt, 2017, Learning with confident examples: rank pruning for robust classification with noisy labels, ArXiv e-prints
- Quekel, 1999, Miss rate of lung cancer on the chest radiograph in clinical practice, Chest, 115, 720, 10.1378/chest.115.3.720 · DOI · Paper page
- Ranzato, 2014, On learning where to look.
- Rensink, 2000, The dynamic representation of scenes, Vis. cogn., 7, 17, 10.1080/135062800394667 · DOI · Paper page
- Sermanet, 2014, Attention for fine-grained categorization.
- Shin, 2016, Learning to read chest x-rays: recurrent neural cascade model for automated image annotation.
- Simonyan, 2014, Very deep convolutional networks for large-scale image recognition.
- Szucs-Farkas, 2013, Comparison of dual-energy subtraction and electronic bone suppression combined with computer-aided detection on chest radiographs: effect on human observers’ performance in nodule detection, Am. Jo. Roentgenol., 200, 1006, 10.2214/AJR.12.8877 · doi:10.2214/ajr.12.8877
- Yao, 2017, Deep learning from noisy image labels with quality embedding
- Zamir, 2016, Fedback networks.
- Zhou, 2016, Learning deep features for discriminative localization, CVPR
Cited by
Attention gated networks: learning to leverage salient regions in medical images
Schlemper, J., Oktay, O., Schaap, M., Heinrich, M., Kainz, B., Glocker, B., & Rueckert, D. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.01.012 · Cited by 2,014
Rajpurkar, P., Irvin, J., Ball, R. L., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C. P., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002686 · Cited by 1,449
Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
van der Velden, B. H. M., Kuijf, H. J., Gilhuijs, K. G. A., & Viergever, M. A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102470 · Cited by 1,328
Deep learning for chest X-ray analysis: A survey
Çallı, E., Sogancioglu, E., van Ginneken, B., van Leeuwen, K. G., & Murphy, K. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102125 · Cited by 506
Attention, please! A survey of neural attention models in deep learning
de Santana Correia, A., & Colombini, E. L. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10148-x · Cited by 298
Metadata
- Authors
- Pesce, E., Joseph Withey, S., Ypsilantis, P.-P., Bakewell, R., Goh, V., & Montana, G.
- Venue
- Medical Image Analysis
- Publisher
- Elsevier BV
- Published
- 2019-04
- Volume
- 53
- Pages
- 26-38
- ISSN
- 1361-8415 (ISSN-L)
- OpenAlex
- W2775469102
- Cited by
- 126
- References
- 19
- FWCI
- 8.52
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- Funders
- National Institute for Health Research
- Biomedical Research Centre
- Guy’s & St Thomas’ NHS Foundation Trust
- King’s College Hospital NHS Foundation Trust
- King’s College London/University College
- Cancer Research UK
- Engineering and Physical Sciences Research Council
- Medical Research Council
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.