Deep convolutional neural network and 3D deformable approach for tissue segmentation in musculoskeletal magnetic resonance imaging
Open accessAbstract
PURPOSE: To describe and evaluate a new fully automated musculoskeletal tissue segmentation method using deep convolutional neural network (CNN) and three-dimensional (3D) simplex deformable modeling to improve the accuracy and efficiency of cartilage and bone segmentation within the knee joint. METHODS: A fully automated segmentation pipeline was built by combining a semantic segmentation CNN and 3D simplex deformable modeling. A CNN technique called SegNet was applied as the core of the segmentation method to perform high resolution pixel-wise multi-class tissue classification. The 3D simplex deformable modeling refined the output from SegNet to preserve the overall shape and maintain a desirable smooth surface for musculoskeletal structure. The fully automated segmentation method was tested using a publicly available knee image data set to compare with currently used state-of-the-art segmentation methods. The fully automated method was also evaluated on two different data sets, which include morphological and quantitative MR images with different tissue contrasts. RESULTS: The proposed fully automated segmentation method provided good segmentation performance with segmentation accuracy superior to most of state-of-the-art methods in the publicly available knee image data set. The method also demonstrated versatile segmentation performance on both morphological and quantitative musculoskeletal MR images with different tissue contrasts and spatial resolutions. CONCLUSION: The study demonstrates that the combined CNN and 3D deformable modeling approach is useful for performing rapid and accurate cartilage and bone segmentation within the knee joint. The CNN has promising potential applications in musculoskeletal imaging. Magn Reson Med 79:2379-2391, 2018. © 2017 International Society for Magnetic Resonance in Medicine.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Applications of deep learning in musculoskeletal imaging remain limited.
“While a survey on deep learning in medical image analysis has shown a wide variety of applications in all imaging subspecialties (13), applications in musculoskeletal imaging remain limited (14–17).”
From Deep learning approach for evaluating knee MR images: achieving high diagnostic performance for cartilage lesion detection. (2018) · cites this paper as [14]
The VGG16 network has proven to be an effective and efficient network of feature extraction in many medical image applications, including tissue segmentation and image contrast conversion.
“More recently, the VGG16 network has proven to be an effective and efficient network of feature extraction in many medical image applications, including tissue segmentation (14,24–26) and image contrast conversion (27,28).”
From Deep learning approach for evaluating knee MR images: achieving high diagnostic performance for cartilage lesion detection. (2018) · cites this paper as [14]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · see citing works
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
Yushkevich, P. A., Piven, J., Hazlett, H. C., Smith, R. G., Ho, S., Gee, J. C., & Gerig, G. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.01.015 · Cited by 8,868 · see citing works
Shin, H.-C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528162 · Cited by 5,759 · see citing works
Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation
Kamnitsas, K., Ledig, C., Newcombe, V. F. J., Simpson, J. P., Kane, A. D., Menon, D. K., Rueckert, D., & Glocker, B. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.10.004 · Cited by 3,498 · cited 6× · see citing works
“Therefore, there is no need for multi-stage training (29) and other supporting aids for region proposals (46) and patch extraction (22,26).”
BackgroundMethodInfluentialConditional Random Fields as Recurrent Neural Networks
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., & Torr, P. H. S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.179 · Cited by 2,386 · cited 1× · see citing works
“Such an example is the integration of a fully connected CRF as an recurrent neural network model on top of a CNN, with the two trained together end-to-end with regular back-propagation algorithm (47).”
MethodWhat is the best multi-stage architecture for object recognition?
Jarrett, K., Kavukcuoglu, K., Ranzato, M. A., & LeCun, Y. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459469 · Cited by 2,153 · see citing works
Automatic Segmentation of MR Brain Images With a Convolutional Neural Network
Moeskops, P., Viergever, M. A., Mendrik, A. M., de Vries, L. S., Benders, M. J. N. L., & Isgum, I. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2548501 · Cited by 861 · see citing works
CNN Features off-the-shelf: an Astounding Baseline for Recognition
Razavian, A. S., Azizpour, H., Sullivan, J., & Carlsson, S. (2014) · arXiv · doi:10.48550/arxiv.1403.6382 · Cited by 607 · see citing works
Brosch, T., Tang, L. Y. W., Yoo, Y., Li, D. K. B., Traboulsee, A., & Tam, R. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2528821 · Cited by 460 · see citing works
Flagged first, then by citations
Research works printed but not linked: 44
- [1]Centers for Disease Control and Prevention. Arthritis prevalence and activity limitations--United States, 1990. JAMA 1992; 272: 346– 347. 8028153
- [2]Felson DT. An update on the pathogenesis and epidemiology of osteoarthritis. Radiol Clin North Am 2004; 42: 1– 9. 15049520
- [3]Oliveria SA, Felson DT, Reed JI, Cirillo PA, Walker AM. Incidence of symptomatic hand, hip, and knee osteoarthritis among patients in a health maintenance organization. Arthritis Rheum 1995; 38: 1134– 1141. 7639811
- [4]Quatman CE, Hettrich CM, Schmitt LC, Spindler KP. The clinical utility and diagnostic performance of magnetic resonance imaging for identification of early and advanced knee osteoarthritis: a systematic review. Am J Sports Med 2011; 39: 1557– 1568. 21730207 PMC3782308
- [5]Conaghan PG, Felson D, Gold G, Lohmander S, Totterman S, Altman R. MRI and non-cartilaginous structures in knee osteoarthritis. Osteo-arthritis Cartilage 2006; 14 Suppl A:A87– 94. 16713722
- [6]Eckstein F, Burstein D, Link TM. Quantitative MRI of cartilage and bone: degenerative changes in osteoarthritis. NMR Biomed 2006; 19: 822– 854. 17075958
- [7]Eckstein F, Cicuttini F, Raynauld JP, Waterton JC, Peterfy C. Magnetic resonance imaging (MRI) of articular cartilage in knee osteoarthritis (OA): morphological assessment. Osteoarthritis Cartilage 2006; 14 Suppl A:A46– 75. 16713720
- [8]McWalter EJ, Wirth W, Siebert M, von Eisenhart-Rothe RM, Hudelmaier M, Wilson DR, Eckstein F. Use of novel interactive input devices for segmentation of articular cartilage from magnetic resonance images. Osteoarthritis Cartilage 2005; 13: 48– 53. 15639637
- [10]Pedoia V, Majumdar S, Link TM. Segmentation of joint and musculo-skeletal tissue in the study of arthritis. MAGMA 2016; 29: 207– 221. 26915082 PMC7181410
- [11]Fripp J, Crozier S, Warfield SK, Ourselin S. Automatic segmentation and quantitative analysis of the articular cartilages from magnetic resonance images of the knee. IEEE Trans Med Imaging 2010; 29: 55– 64. 19520633 PMC3717377
- [12]Seim H, Kainmueller D, Lamecker H, Bindernagel M, Malinowski J, Zachow S. Model-based auto-segmentation of knee bones and cartilage in mri data. Medical Image Analysis for the Clinic: a Grand Challenge. In Proceedings of the 13th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2010), Beijing, China, 2010 p 215– 223.
- [13]Vincent G, Wolstenholme C, Scott I, Bowes M. Fully automatic segmentation of the knee joint using active appearance models. Medical Image Analysis for the Clinic: a Grand Challenge. In Proceedings of the 13th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2010), Beijing, China, 2010 p 224– 230.
- [14]Tamez-Pena JG, Farber J, Gonzalez PC, Schreyer E, Schneider E, Totterman S. Unsupervised segmentation and quantification of anatomical knee features: data from the osteoarthritis initiative. IEEE Trans Biomed Eng 2012; 59: 1177– 1186. 22318477
- [15]Dam EB, Lillholm M, Marques J, Nielsen M. Automatic segmentation of high- and low-field knee MRIs using knee image quantification with data from the osteoarthritis initiative. J Med Imaging (Bellingham) 2015; 2: 024001. 26158096 PMC4478858
- [16]Piplani MA, Disler DG, McCauley TR, Holmes TJ, Cousins JP. Articular cartilage volume in the knee: semiautomated determination from three-dimensional reformations of MR images. Radiology 1996; 198: 855– 859. 8628883
- [17]Gougoutas AJ, Wheaton AJ, Borthakur A, Shapiro EM, Kneeland JB, Udupa JK, Reddy R. Cartilage volume quantification via Live Wire segmentation. Acad Radiol 2004; 11: 1389– 1395. 15596377
- [18]Kshirsagar AA, Watson PJ, Tyler JA, Hall LD. Measurement of localized cartilage volume and thickness of human knee joints by computer analysis of three-dimensional magnetic resonance images. Invest Radiol 1998; 33: 289– 299. 9609488
- [19]Stammberger T, Eckstein F, Michaelis M, Englmeier KH, Reiser M. Interobserver reproducibility of quantitative cartilage measurements: comparison of B-spline snakes and manual segmentation. Magn Reson Imaging 1999; 17: 1033– 1042. 10463654
- [20]Heimann T, Morrison B, Styner M, Niethammer M, Warfield S. Segmentation of knee images: a grand challenge. Medical Image Analysis for the Clinic: a Grand Challenge. In Proceedings of the 13th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2010), Beijing, China, 2010 p 207– 214.
- [21]Greenspan HG, Bram van, Summers RM. Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans Med Imaging 2016; 35: 1153– 1159.
- [22]Pereira S, Pinto A, Alves V, Silva CA. Brain tumor segmentation using convolutional neural networks in mri images. IEEE Trans Med Imaging 2016; 35: 1240– 1251. 26960222
- [25]Avendi MR, Kheradvar A, Jafarkhani H. A combined deep-learning and deformable-model approach to fully automatic segmentation of the left ventricle in cardiac MRI. Med Image Anal 2016; 30: 108– 119. 26917105
- [27]Antony J, McGuinness K, Connor NEO, Moran K. Quantifying radio-graphic knee osteoarthritis severity using deep convolutional neural networks. arXiv preprint arXive: 1609.02469.
- [28]Prasoon A, Petersen K, Igel C, Lauze F, Dam E, Nielsen M. Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. Med Image Comput Comput Assist Interv 2013; 16(Pt 2): 246– 253. 24579147
- [29]Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. arXiv preprint arXiv: 1605.06211. 27244717 · arXiv
- [32]Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409.1556. · arXiv
- [33]Delingette H General object reconstruction based on simplex meshes. Int J Comput Vision 1999; 32: 111– 146.
- [34]Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T. Caffe: Convolutional architecture for fast feature embedding. arXiv preprints arXive 2014: 14082014.
- [35]Ioffe S, Szegedy C. Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 1502.03167. · arXiv
- [36]Nair V, Hinton GE. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on Machine Learning, Haifa, Israel, 2010 p 807– 814.
- [37]Lorensen WE, Cline HE. Marching cubes: a high resolution 3D surface construction algorithm. SIGGRAPH Comput Graph 1987; 21: 163– 169.
- [38]Perona P, Malik J. Scale-space and edge detection using anisotropic diffusion. IEEE Trans Pattern Anal Mach Intell 1990; 12: 629– 639.
- [39]Young IT, van Vliet LJ. Recursive implementation of the Gaussian filter. Signal Process 1995; 44: 139– 151.
- [40]Cohen LD, Cohen I. A finite element method applied to new active contour models and 3D reconstruction from cross sections. In Proceedings of the 3rd International Conference on Computer Vision, Osaka, Japan, 1990 p 587– 591.
- [41]Zarins ZA, Bolbos RI, Pialat JB, Link TM, Li X, Souza RB, Majumdar S. Cartilage and meniscus assessment using T1rho and T2 measurements in healthy subjects and patients with osteoarthritis. Osteoarthritis Cartilage 2010; 18: 1408– 1416. 20696262 PMC2975868
- [42]Stehling C, Baum T, Mueller-Hoecker C, Liebl H, Carballido-Gamio J, Joseph GB, Majumdar S, Link TM. A novel fast knee cartilage segmentation technique for T2 measurements at MR imaging--data from the Osteoarthritis Initiative. Osteoarthritis Cartilage 2011; 19: 984– 989. 21515391 PMC6338671
- [45]Bottou L Large-scale machine learning with stochastic gradient descent. In Proceedings of the 19th International Conference on Computational Statistics, Heidelberg, Germany, 2010 p 177– 186.
- [46]Noh H, Hong S, Han B. Learning deconvolution network for semantic segmentation. arXiv preprint arXiv: 1505.04366. · arXiv
- [48]Chen LC, Papandreou G, Kokkinos I, Murphy K, Yuille AL. Semantic image segmentation with deep convolutional nets and fully connected CRFs. arXiv preprint arXiv: 1412.7062. 28463186 · arXiv
- [49]Shin HC, Le L, Kim L, Seff A, Yao J, Summers RM. Interleaved text/image deep mining on a large-scale radiology database. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, Massachusetts, USA, 2015 p 1090– 1099.
- [50]Girshick R, Donahue J, Darrell T, Malik J. Region-based convolutional networks for accurate object detection and segmentation. IEEE Trans Pattern Anal Mach Intell 2016; 38: 142– 158. 26656583
- [53]Wang Z, Donoghue C, Rueckert D. Patch-based segmentation without registration: application to knee MRI In: Wu G, Zhang D, Shen D, Yan P, Suzuki K, Wang F, editors. Machine learning in medical imaging: 4th international workshop, MLMI 2013. Cham: Springer International Publishing; 2013 p 98– 105.
- [54]Lee S, Shim H, Park SH, Yun ID, Lee SU. Learning local shape and appearance for segmentation of knee cartilage in 3D MRI. Medical Image Analysis for the Clinic: a Grand Challenge. In Proceedings of the 13th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2010), Beijing, China, 2010 p 231– 240.
- [55]Yin Y, Williams R, Anderson DD, Sonka M. Hierarchical decision framework with a priori shape models for knee joint cartilage segmentation. Medical Image Analysis for the Clinic: a Grand Challenge. In Proceedings of the 13th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2010), Beijing, China, 2010 p 241– 250.
Cited by
Sorted by citations
Liu, F., Zhou, Z., Samsonov, A., Blankenbaker, D., Larison, W., Kanarek, A., Lian, K., Kambhampati, S., & Kijowski, R. (2018) · Radiology · doi:10.1148/radiol.2018172986 · Cited by 297 · see citing works
“The segmentation CNN in the processing pipeline was constructed by using a 2D convolutional encoder-decoder network that has been previously shown to provide rapid and accurate segmentation of cartilage and bone on knee MR images (14).”
Self-citationYu, K., Ying, J., Zhao, T., Lei, L., Zhong, L., Hu, J., Zhou, J. W., Huang, C., & Zhang, X. (2023) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-22-368 · Cited by 34 · see citing works
Deep learning in rheumatological image interpretation
Stoel, B. C., Staring, M., Reijnierse, M., & van der Helm-van Mil, A. H. M. (2024) · Nature Reviews Rheumatology · doi:10.1038/s41584-023-01074-5 · Cited by 25 · see citing works
- Authors
- Liu, F., Zhou, Z., Jang, H., Samsonov, A., Zhao, G., & Kijowski, R.
- Affiliations
- University of Wisconsin–Madison (US)
- University of Minnesota (US)
- Corresponding author
- Liu, Fang
- Venue
- Magnetic Resonance in Medicine
- Publisher
- Wiley
- Published
- 2017-07-21
- Volume
- 79
- Issue
- 4
- Pages
- 2379-2391
- ISSN
- 0740-3194 (ISSN-L)
- 1522-2594
- PMC
- PMC6271435
- OpenAlex
- W2737373222
- Cited by
- 319
- References
- 11
- Citations by year
- 2026: 11 · 2025: 29 · 2024: 21 · 2023: 37 · 2022: 46 · 2021: 42 · 2020: 46 · 2019: 53 · 2018: 30 · 2017: 3 · 2013: 1
- FWCI
- 14.98
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Magnetic Resonance in Medicine (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institute of Arthritis and Musculoskeletal and Skin Diseases (R01AR068373)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Engineering › Biomedical Engineering › Medical Imaging and Analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- knee joint, cartilage segmentation, bone segmentation, deep convolutional neural networks, SegNet, semantic segmentation, quantitative MRI, automated image segmentation
- MeSH terms
- Algorithms, Bone and Bones, Cartilage, Humans, Image Processing, Computer-Assisted, Knee, Magnetic Resonance Imaging, Pattern Recognition, Automated, Software, Reproducibility of Results, Neural Networks, Computer, Imaging, Three-Dimensional
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
