Automated Muscle Segmentation from Clinical CT Using Bayesian U-Net for Personalized Musculoskeletal Modeling
Open accessThe DOI matches its Crossref record; 43 of the 45 references it cites were found in a registry or an index.
Abstract
We propose a method for automatic segmentation of individual muscles from a clinical CT. The method uses Bayesian convolutional neural networks with the U-Net architecture, using Monte Carlo dropout that infers an uncertainty metric in addition to the segmentation label. We evaluated the performance of the proposed method using two data sets: 20 fully annotated CTs of the hip and thigh regions and 18 partially annotated CTs that are publicly available from The Cancer Imaging Archive (TCIA) database. The experiments showed a Dice coefficient (DC) of 0.891±0.016 (mean±std) and an average symmetric surface distance (ASD) of 0.994±0.230 mm over 19 muscles in the set of 20 CTs. These results were statistically significant improvements compared to the state-of-the-art hierarchical multi-atlas method which resulted in 0.845 ± 0.031 DC and 1.556 ± 0.444 mm ASD. We evaluated validity of the uncertainty metric in the multi-class organ segmentation problem and demonstrated a correlation between the pixels with high uncertainty and the segmentation failure. One application of the uncertainty metric in active-learning is demonstrated, and the proposed query pixel selection method considerably reduced the manual annotation cost for expanding the training data set. The proposed method allows an accurate patient-specific analysis of individual muscle shapes in a clinical routine. This would open up various applications including personalization of biomechanical simulation and quantitative evaluation of muscle atrophy.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Xue al. [685] | et | 2019 | BNN and U-Net | Classification | Gigapixel phase im- ages | 1 | MC dropout and ensembles”
From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [685]
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 4×
“the weights in the same way as in [9], and then trained the networks using adaptive moment estimation (Adam) [16] for 1×105 iterations at the learning rate of 0.”
MethodInfluentialFully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
(2014) · Cited by 33,419
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741 · cited 1×
“We initialized the weights in the same way as in [9], and then trained the networks using adaptive moment estimation (Adam) [16] for 1 × 10 5 iterations at the learning rate of 0.0001.”
MethodInfluentialMeasures of the Amount of Ecologic Association Between Species
Dice, L. R. (1945) · Ecology · doi:10.2307/1932409 · Cited by 11,874 · cited 1×
“We used the Dice coefficient (DC) [5] and the average symmetric surface distance (ASD) [31] as the error metrics.”
MethodDropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471 · cited 6×
“Below, we briefly summarize the theoretical background described in [7], formulate the specific metric that we employed in this paper, and propose a new structure-wise”
BackgroundMethodInfluential(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088 · cited 1×
“It measures the degree of difference of each test sample from the training data set, originated from the deficiency of training data, namely epistemic uncertainty [15].”
BackgroundSeth, A., Hicks, J. L., Uchida, T. K., Habib, A., Dembia, C. L., Dunne, J. J., Ong, C. F., DeMers, M. S., Rajagopal, A., Millard, M., et al. (2018) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1006223 · Cited by 1,322 · cited 1×
“The computational simulation of human motion using musculoskeletal modeling has been performed in a number of studies to investigate musculo-tendon forces and joint contact forces, which cannot be easily achieved by physical measurements [25], [27], [29].”
BackgroundFull-Body Musculoskeletal Model for Muscle-Driven Simulation of Human Gait
Rajagopal, A., Dembia, C. L., DeMers, M. S., Delp, D. D., Hicks, J. L., & Delp, S. L. (2016) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2586891 · Cited by 1,053
Vallières, M., Freeman, C. R., Skamene, S. R., & El Naqa, I. (2015) · Physics in Medicine and Biology · doi:10.1088/0031-9155/60/14/5471 · Cited by 897
Kendall, A., Badrinarayanan, V., & Cipolla, R. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.57 · Cited by 784
Multi-atlas segmentation of biomedical images: A survey
Iglesias, J. E., & Sabuncu, M. R. (2015) · Medical Image Analysis · doi:10.1016/j.media.2015.06.012 · Cited by 667
Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentation
Yang, L., Zhang, Y., Chen, J., Zhang, S., & Chen, D. Z. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-66179-7_46 · Cited by 480 · cited 6×
“[39] which utilized uncertainty and similarity metrics to determine the query slices.”
BackgroundMethodInfluentialQuantitative computed tomography
Adams, J. E. (2009) · European Journal of Radiology · doi:10.1016/j.ejrad.2009.04.074 · Cited by 458 · cited 4×
“CT (QCT) [1] which is one of our main application targets for the analysis of the relationship between muscle quality and bone mineral density (see supplementary materials for evaluation of the skin surface segmentation step in QCT volumes).”
BackgroundInfluentialNair, T., Precup, D., Arnold, D. L., & Arbel, T. (2020) · Medical Image Analysis · doi:10.1016/j.media.2019.101557 · Cited by 451
Relationships of 35 lower limb muscles to height and body mass quantified using MRI
Handsfield, G. G., Meyer, C. H., Hart, J. M., Abel, M. F., & Blemker, S. S. (2014) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2013.12.002 · Cited by 412 · cited 1×
“Handsfield et al. [8] manually performed segmentation of 35 individual muscles from MRIs of the lower leg in order to investigate the relationship between muscle volume and height or weight.”
MethodDeep Adversarial Networks for Biomedical Image Segmentation Utilizing Unannotated Images
Zhang, Y., Yang, L., Chen, J., Fredericksen, M., Hughes, D. P., & Chen, D. Z. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-66179-7_47 · Cited by 379 · cited 1×
“be achievable, for example, by using an algorithm such as CycleGAN [11], [41] for synthesizing a CT-like image from”
MethodKendall, A., Badrinarayanan, V., & Cipolla, R. (2015) · arXiv · doi:10.48550/arxiv.1511.02680 · Cited by 292
Hiasa, Y., Otake, Y., Takao, M., Matsuoka, T., Takashima, K., Carass, A., Prince, J. L., Sugano, N., & Sato, Y. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-00536-8_4 · Cited by 221
Self-citationKarlsson, A., Rosander, J., Romu, T., Tallberg, J., Grönqvist, A., Borga, M., & Dahlqvist Leinhard, O. (2015) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.24726 · Cited by 219 · cited 1×
“[13] implemented an algorithm for automated segmentation of the muscle and fat tissues from MRI using a multi-atlas method [12].”
Method3D Segmentation in the Clinic: A Grand Challenge II: MS lesion segmentation
Styner, M., Lee, J., Chin, B., Chin, M. S., Commowick, O., Tran, H.-H., Markovic-Plese, S., Jewells, V., & Warfield, S. (2008) · The MIDAS Journal · doi:10.54294/lmkqvm · Cited by 167 · cited 13×
“When excluding four outlier patients with extremely large sarcoma, the average values of DC and ASD were 0.925 ± 0.014 and 1.135 ± 0.777 mm, respectively, which was comparable to the results on the THA data set.”
MethodInfluentialLee, H., Troschel, F. M., Tajmir, S., Fuchs, G., Mario, J., Fintelmann, F. J., & Do, S. (2017) · Journal of Digital Imaging · doi:10.1007/s10278-017-9988-z · Cited by 163 · cited 1×
“Lee et al. [17] used deep learning for segmentation of the muscle and fat tissues in a 2D abdominal CT slice.”
BackgroundPersisting muscle atrophy two years after replacement of the hip
Rasch, A., Byström, A. H., Dalén, N., Martinez-Carranza, N., & Berg, H. E. (2009) · The Journal of Bone and Joint Surgery. British volume · doi:10.1302/0301-620x.91b5.21477 · Cited by 160 · cited 1×
“disuse, malnutrition, and diabetes [26], [33], where muscles were delineated manually by a single operator from the”
BackgroundReverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth
Valindria, V. V., Lavdas, I., Bai, W., Kamnitsas, K., Aboagye, E. O., Rockall, A. G., Rueckert, D., & Glocker, B. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2665165 · Cited by 152 · cited 2×
“Two example applications of the uncertainty metric explored in this study are; 1) prediction of segmentation accuracy without using the ground truth similar to the goal of Valindria et al. [35] and, 2) the active-learning framework [19], [39] for the reduction of manual annotation costs.”
MethodEaton-Rosen, Z., Bragman, F., Bisdas, S., Ourselin, S., & Cardoso, M. J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00928-1_78 · Cited by 108
3D finite element models of shoulder muscles for computing lines of actions and moment arms
Webb, J. D., Blemker, S. S., & Delp, S. L. (2012) · Computer Methods in Biomechanics & Biomedical Engineering · doi:10.1080/10255842.2012.719605 · Cited by 97
Nair, T., Precup, D., Arnold, D. L., & Arbel, T. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00928-1_74 · Cited by 92
Prior Knowledge, Random Walks and Human Skeletal Muscle Segmentation
Baudin, P., Azzabou, N., Carlier, P. G., & Nikos Paragios, A. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33415-3_70 · Cited by 78 · cited 1×
“To facilitate automation of the individual muscular segmentation, prior knowledge about the shape of each muscle has been introduced [3].”
MethodShu, L., Yamamoto, K., Jiang, Y. D., Saraswat, P., Liu, Y., Mitsuishi, M., & Sugita, N. (2018) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2018.07.008 · Cited by 64 · cited 1×
“putational simulation of human motion using musculoskeletal modeling has been performed in a number of studies to investigate musculo-tendon forces and joint contact forces, which cannot be easily achieved by physical measurements [25], [27], [29].”
BackgroundMoissenet, F., Modenese, L., & Dumas, R. (2017) · Journal of Biomechanics · doi:10.1016/j.jbiomech.2017.08.025 · Cited by 63 · cited 1×
“Recent studies have demonstrated that personalization of model parameters, such as the size of the bones, geometry of the muscles and tendons, and physical properties of the muscle-tendon complex, improves accuracy of the simulation [4], [20], [32].”
BackgroundAndrews, S., & Hamarneh, G. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2403299 · Cited by 57 · cited 2×
“The target region was broader than [2] covering the origin to insertion of 19 muscles.”
BackgroundMethodVolume Increases of the Gluteus Maximus, Gluteus Medius, and Thigh Muscles After Hip Arthroplasty
Uemura, K., Takao, M., Sakai, T., Nishii, T., & Sugano, N. (2015) · The Journal of Arthroplasty · doi:10.1016/j.arth.2015.10.036 · Cited by 56 · cited 1×
“disuse, malnutrition, and diabetes [26], [33], where muscles were delineated manually by a single operator from the”
BackgroundSelf-citationCrowd-Algorithm Collaboration for Large-Scale Endoscopic Image Annotation with Confidence
Maier‐Hein, L., Ross, T., Gröhl, J., Glocker, B., Bodenstedt, S., Stock, C., Heim, E., GOETZ, M. A., Wirkert, S., Kenngott, H., et al. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46723-8_71 · Cited by 46 · cited 4×
“In summary, the proposed method combines slice- [39] and pixel- [19] selection methods based on Bayesian neural networks [7].”
BackgroundMethodInfluentialFemoral loads during gait in a patient with massive skeletal reconstruction
Taddei, F., Martelli, S., Valente, G., Leardini, A., Benedetti, M. G., Manfrini, M., & Viceconti, M. (2011) · Clinical Biomechanics · doi:10.1016/j.clinbiomech.2011.09.006 · Cited by 40 · cited 1×
“Recent studies have demonstrated that personalization of model parameters, such as the size of the bones, geometry of the muscles and tendons, and physical properties of the muscle-tendon complex, improves accuracy of the simulation [4], [20], [32].”
BackgroundUlbrich, E. J., Nanz, D., Leinhard, O. D., Marcon, M., & Fischer, M. A. (2018) · Magnetic Resonance in Medicine · doi:10.1002/mrm.26676 · Cited by 38 · cited 1×
“Ulbrich et al. [34] and Karlsson et al. [13] implemented an algorithm for automated segmentation of the muscle and fat tissues from MRI using a multi-atlas method [12].”
MethodFutoshi, Y., Otake, Y., Takao, M., Ogawa, T., Okada, T., Sugano, N., & Sato, Y. (2018) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-018-1758-y · Cited by 36 · cited 6×
“Heatmap visualization of ASD with hierarchical multi-atlas method [40] and U-Net for each individual muscle in each patient.”
BackgroundMethodInfluentialSelf-citationOgawa, T., Takao, M., Otake, Y., Futoshi, Y., Hamada, H., Sakai, T., Sato, Y., & Sugano, N. (2019) · Journal of Orthopaedic Science · doi:10.1016/j.jos.2019.02.004 · Cited by 30 · cited 13×
“When excluding four outlier patients with extremely large sarcoma, the average values of DC and ASD were 0.925 ± 0.014 and 1.135 ± 0.777 mm, respectively, which was comparable to the results on the THA data set.”
MethodInfluentialSelf-citationState of the art and current limits of musculo-skeletal models for clinical applications
Chèze, L. L., Moissenet, F., & Dumas, R. (2012) · Movement & Sport Sciences - Science & Motricité · doi:10.1051/sm/2012026 · Cited by 29 · cited 1×
“Recent studies have demonstrated that personalization of model parameters, such as the size of the bones, geometry of the muscles and tendons, and physical properties of the muscle-tendon complex, improves accuracy of the simulation [4], [20], [32].”
BackgroundMurai, A., Endo, Y., & Tada, M. (2016) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2016.2524069 · Cited by 19 · cited 1×
“The volumetric muscle modeling, using a finite element model [37], [38] or a simpler approximation in shape deformation for real-time applications such as [21], has shown advantages in accurate prediction of muscle behavior.”
MethodYamamura, N., Alves, J. L., Oda, T., Kinugasa, R., & Takagi, S. (2014) · Journal of Biomechanical Science and Engineering · doi:10.1299/jbse.13-00294 · Cited by 10 · cited 1×
“The volumetric muscle modeling, using a finite element model [37], [38] or a simpler approximation in shape deformation for real-time applications such as [21], has shown advantages in accurate prediction of muscle behavior.”
MethodEaton-Rosen, Z., Bragman, F., Bisdas, S., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.08640 · Cited by 9
Otake, Y., Takao, M., Fukuda, N., Takagi, S., Yamamura, N., Sugano, N., & Sato, Y. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00928-1_79 · Cited by 3 · cited 1×
“In addition, Otake et al. [24] demonstrated the potential for estimating the patient-specific muscle fiber structure from a clinical CT assuming the segmentation of each muscle was provided.”
BackgroundSelf-citation
Research works printed but not linked: 2
- hiasa, 2018, Surgical tools segmentation in laparoscopic images using convolutional neural networks with uncertainty estimation and semi-supervised learning, Proc Int Conf CARS, 14
- hiasa, 2018, Cross-modality image synthesis from unpaired data using CycleGAN, Proc Int Workshop SASHIMI, 31
Cited by
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111
U-Net and Its Variants for Medical Image Segmentation: A Review of Theory and Applications
Siddique, N., Paheding, S., Elkin, C. P., & Devabhaktuni, V. (2021) · IEEE Access · doi:10.1109/access.2021.3086020 · Cited by 1,960
Lambert, B., Forbes, F., Doyle, S., Dehaene, H., & Dojat, M. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102830 · Cited by 269
Modality specific U-Net variants for biomedical image segmentation: a survey
Punn, N. S., & Agarwal, S. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10152-1 · Cited by 246
“(2018) Improved U-Net CT - X X - BU Mini-residual connections within encoder-decoder phases Janssens et al. (2018) Two stage U-Net model CT - X X - EU 3D FCN LocalizationNet followed by SegmentationNet Kumar et al. (2018) U-SegNet MRI X - X - BU Integration of skip connections with SegNet Kermi et al. (2018) Residual…”
MethodInfluentialSeoni, S., Jahmunah, V., Salvi, M., Barua, P. D., Molinari, F., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107441 · Cited by 239
Metadata
- Authors
- Hiasa, Y., Otake, Y., Takao, M., Ogawa, T., Sugano, N., & Sato, Y.
- Affiliations
- Nara Institute of Science and Technology (JP)
- The University of Osaka (JP)
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2020-04
- Volume
- 39
- Issue
- 4
- Pages
- 1030-1040
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2972629588
- Cited by
- 157
- References
- 45
- Citations by year
- 2026: 16 · 2025: 27 · 2024: 26 · 2023: 24 · 2022: 26 · 2021: 22 · 2020: 16
- FWCI
- 7.16
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Ministry of Education, Culture, Sports, Science and Technology (19H01176, 26108004)
- Japan Society for the Promotion of Science (20407)
- AMED/ETH Strategic Japanese-Swiss Cooperative Program
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Engineering › Biomedical Engineering › Medical Imaging and Analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- muscle segmentation, Monte Carlo dropout, uncertainty estimation, musculoskeletal modeling, active learning (machine learning), muscle atrophy, biomechanical simulation, multi-atlas segmentation
- MeSH terms
- Deep Learning, Bayes Theorem, Female, Hip, Humans, Image Processing, Computer-Assisted, Male, Thigh, Tomography, X-Ray Computed, Neural Networks, Computer, Muscle, Skeletal, Arthroplasty, Replacement, Hip, Patient-Specific Modeling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.