Performance of a Deep-Learning Neural Network Model in Assessing Skeletal Maturity on Pediatric Hand Radiographs
Abstract
Purpose To compare the performance of a deep-learning bone age assessment model based on hand radiographs with that of expert radiologists and that of existing automated models. Materials and Methods The institutional review board approved the study. A total of 14 036 clinical hand radiographs and corresponding reports were obtained from two children’s hospitals to train and validate the model. For the first test set, composed of 200 examinations, the mean of bone age estimates from the clinical report and three additional human reviewers was used as the reference standard. Overall model performance was assessed by comparing the root mean square (RMS) and mean absolute difference (MAD) between the model estimates and the reference standard bone ages. Ninety-five percent limits of agreement were calculated in a pairwise fashion for all reviewers and the model. The RMS of a second test set composed of 913 examinations from the publicly available Digital Hand Atlas was compared with published reports of an existing automated model. Results The mean difference between bone age estimates of the model and of the reviewers was 0 years, with a mean RMS and MAD of 0.63 and 0.50 years, respectively. The estimates of the model, the clinical report, and the three reviewers were within the 95% limits of agreement. RMS for the Digital Hand Atlas data set was 0.73 years, compared with 0.61 years of a previously reported model. Conclusion A deep-learning convolutional neural network model can estimate skeletal maturity with accuracy similar to that of an expert radiologist and to that of existing automated models. © RSNA, 2017 An earlier incorrect version of this article appeared online. This article was corrected on January 19, 2018.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Deep learning models have demonstrated diagnostic accuracy comparable to that of human experts in narrow clinical tasks for several medical domains and imaging modalities.
“deep learning models have demonstrated diagnostic accuracy comparable to that of human experts in narrow clinical tasks for several medical domains and imaging modalities, including chest and extremity X-rays8–10, computed tomography (CT)11, magnetic resonance imaging (MRI)12, whole slide images (WSI)13,14, and dermatology images15.”
From Self-supervised learning for medical image classification: a systematic review and implementation guidelines (2023) · cites this paper as [10]
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · see citing works
RADIOGRAPHIC ATLAS OF SKELETAL DEVELOPMENT OF THE HAND AND WRIST
GREULICH, W. W., & Pyle, S. I. (1959) · The American Journal of the Medical Sciences · doi:10.1097/00000441-195909000-00030 · Cited by 5,598 · see citing works
Contrast Limited Adaptive Histogram Equalization
Zuiderveld, K. (1994) · Graphics Gems · doi:10.1016/b978-0-12-336156-1.50061-6 · Cited by 4,192 · see citing works
Predicting the future - big data, machine learning, and clinical medicine
Obermeyer, Z., & Emanuel, E. J. (2016) · New England Journal of Medicine · doi:10.1056/nejmp1606181 · Cited by 3,652 · see citing works
Greenspan, H. K., van Ginneken, B., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2553401 · Cited by 1,704 · see citing works
Machine learning for medical imaging
Erickson, B. J., Korfiatis, P., Akkus, Z., & Kline, T. L. (2017) · RadioGraphics · doi:10.1148/rg.2017160130 · Cited by 1,672 · see citing works
Assessment of Skeletal Maturity and Prediction of Adult Height (TW2 Method)
(1976) · Archives of Disease in Childhood · doi:10.1136/adc.51.6.488 · Cited by 1,413 · see citing works
The BoneXpert Method for Automated Determination of Skeletal Maturity
Thodberg, H. H., Kreiborg, S., Juul, A., & Pedersen, K. D. (2009) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2008.926067 · Cited by 438 · see citing works
Implementing Machine Learning in Radiology Practice and Research
Kohli, M. D., Prevedello, L. M. S., Filice, R. W., & Geis, J. R. (2017) · American Journal of Roentgenology · doi:10.2214/ajr.16.17224 · Cited by 301 · see citing works
Bull, R. K., Edwards, P. D., Kemp, P. M., Fry, S., & Hughes, I. A. (1999) · Archives of Disease in Childhood · doi:10.1136/adc.81.2.172 · Cited by 248 · see citing works
Bone age assessment of children using a digital hand atlas
Gertych, A., Zhang, A., Sayre, J. W., et al. (2007) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2007.02.012 · Cited by 217 · see citing works
Computer-assisted bone age assessment: image preprocessing and epiphyseal/metaphyseal ROI extraction
Piętka, E., Gertych, A., Pospiech, S., et al. (2001) · IEEE Transactions on Medical Imaging · doi:10.1109/42.938240 · Cited by 209 · see citing works
Racial Differences in Growth Patterns of Children Assessed on the Basis of Bone Age
Zhang, A., Sayre, J. W., Vachon, L. A., et al. (2008) · Radiology · doi:10.1148/radiol.2493080468 · Cited by 191 · see citing works
HANDX: a model-based system for automatic segmentation of bones from digital hand radiographs
Michael, D. J., & Nelson, A. C. (1989) · IEEE Transactions on Medical Imaging · doi:10.1109/42.20363 · Cited by 120 · see citing works
Groell, R., Lindbichler, F., Riepl, T., Gherra, L., Roposch, A., & Fotter, R. (1999) · British Journal of Radiology · doi:10.1259/bjr.72.857.10505010 · Cited by 107 · see citing works
Computer-assisted phalangeal analysis in skeletal age assessment
Piętka, E., McNitt‐Gray, M. F., Kuo, M. L., et al. (1991) · IEEE Transactions on Medical Imaging · doi:10.1109/42.108597 · Cited by 84 · see citing works
Bone age assessment: automated techniques coming of age?
van Rijn, R. R., & Thodberg, H. H. (2013) · Acta Radiologica · doi:10.1258/ar.2012.120443 · Cited by 64 · see citing works
EFFECT OF TRAINING ON REPLICABILITY OF ASSESSMENTS OF SKELETAL MATURITY (GREULICH-PYLE)
Roche, A. F., Rohmann, C. G., French, N. Y., et al. (1970) · American Journal of Roentgenology · doi:10.2214/ajr.108.3.511 · Cited by 62 · see citing works
A Computerized Image Analysis System for Estimating Tanner-Whitehouse 2 Bone Age
Tanner, J. M., & Gibbons, R. D. (2008) · Hormone Research · doi:10.1159/000184210 · Cited by 56 · see citing works
Automatic Bone Age Measurement Using Computerized Image Analysis
Tanner, J. M., & Gibbons, R. D. (1994) · Journal of Pediatric Endocrinology and Metabolism · doi:10.1515/jpem.1994.7.2.141 · Cited by 56 · see citing works
Breast Tissue Segmentation and Mammographic Risk Scoring Using Deep Learning
Petersen, K., Nielsen, M., Diao, P., et al. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-07887-8_13 · Cited by 52 · see citing works
RELIABILITY OF SKELETAL AGE ASSESSMENTS
Johnson, G., Dorst, J. P., Kuhn, J. P., et al. (1973) · American Journal of Roentgenology · doi:10.2214/ajr.118.2.320 · Cited by 46 · see citing works
Frisch, H., Riedl, S., & Waldhör, T. (1996) · Pediatric Radiology · doi:10.1007/bf01405306 · Cited by 39 · see citing works
Validation of Bone Age Methods by Their Ability to Predict Adult Height
Thodberg, H. H., Neuhof, J., Ranke, M. B., et al. (2010) · Hormone Research in Paediatrics · doi:10.1159/000313592 · Cited by 38 · see citing works
Breen, M. A., Tsai, A., Stamm, A., et al. (2016) · Pediatric Radiology · doi:10.1007/s00247-016-3618-7 · Cited by 36 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Preparing Medical Imaging Data for Machine Learning
Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., & Lungren, M. P. (2020) · Radiology · doi:10.1148/radiol.2020192224 · Cited by 1,020 · see citing works
Self-citationMazurowski, M. A., Buda, M., Saha, A., & Bashir, M. R. (2019) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.26534 · Cited by 554 · see citing works
Huang, S.-C., Pareek, A., Jensen, M., Lungren, M. P., Yeung, S., & Chaudhari, A. S. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00811-0 · Cited by 479 · see citing works
“…accuracy comparable to that of human experts in narrow clinical tasks for several medical domains and imaging modalities, including chest and extremity X-rays 8 – 10 , computed tomography (CT) 11 , magnetic resonance imaging (MRI) 12 , whole slide images (WSI) 13,14 , and dermatology images 15 .”
BackgroundSelf-citationDeep learning with convolutional neural network in radiology
Yasaka, K., Akai, H., Kunimatsu, A., Kiryu, S., & Abe, O. (2018) · Japanese Journal of Radiology · doi:10.1007/s11604-018-0726-3 · Cited by 368 · see citing works
Assessment of Convolutional Neural Networks for Automated Classification of Chest Radiographs
Dunnmon, J. A., Yi, D., Langlotz, C. P., Ré, C., Rubin, D. L., & Lungren, M. P. (2019) · Radiology · doi:10.1148/radiol.2018181422 · Cited by 206 · see citing works
Self-citationTechnical and clinical overview of deep learning in radiology
Ueda, D., Shimazaki, A., & Miki, Y. (2019) · Japanese Journal of Radiology · doi:10.1007/s11604-018-0795-3 · Cited by 100 · see citing works
- Authors
- Larson, D. B., Chen, M. C., Lungren, M. P., Halabi, S. S., Stence, N. V., & Langlotz, C. P.
- Affiliations
- Children's Hospital Colorado (US)
- Venue
- Radiology
- Published
- 2018-04
- Volume
- 287
- Issue
- 1
- Pages
- 313-322
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2766766852
- Cited by
- 456
- References
- 28
- Citations by year
- 2026: 26 · 2025: 33 · 2024: 48 · 2023: 52 · 2022: 64 · 2021: 58 · 2020: 69 · 2019: 64 · 2018: 40 · 2017: 1
- FWCI
- 276.86
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Arts and Humanities › Archeology › Forensic Anthropology and Bioarchaeology Studies
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Engineering › Biomedical Engineering › Medical Imaging and Analysis
- Keywords
- skeletal maturity assessment, bone age assessment, hand radiographs, pediatric radiology, convolutional neural networks, deep learning, radiologist comparison
- MeSH terms
- Machine Learning, Adolescent, Adult, Age Determination by Skeleton, Child, Child, Preschool, Female, Hand, Humans, Infant, Male, Radiography, Neural Networks, Computer, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
