Artificial Intelligence Algorithm Improves Radiologist Performance in Skeletal Age Assessment: A Prospective Multicenter Randomized Controlled Trial
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Background Previous studies suggest that use of artificial intelligence (AI) algorithms as diagnostic aids may improve the quality of skeletal age assessment, though these studies lack evidence from clinical practice. Purpose To compare the accuracy and interpretation time of skeletal age assessment on hand radiograph examinations with and without the use of an AI algorithm as a diagnostic aid. Materials and Methods In this prospective randomized controlled trial, the accuracy of skeletal age assessment on hand radiograph examinations was performed with (n = 792) and without (n = 739) the AI algorithm as a diagnostic aid. For examinations with the AI algorithm, the radiologist was shown the AI interpretation as part of their routine clinical work and was permitted to accept or modify it. Hand radiographs were interpreted by 93 radiologists from six centers. The primary efficacy outcome was the mean absolute difference between the skeletal age dictated into the radiologists’ signed report and the average interpretation of a panel of four radiologists not using a diagnostic aid. The secondary outcome was the interpretation time. A linear mixed-effects regression model with random center- and radiologist-level effects was used to compare the two experimental groups. Results Overall mean absolute difference was lower when radiologists used the AI algorithm compared with when they did not (5.36 months vs 5.95 months; P = .04). The proportions at which the absolute difference exceeded 12 months (9.3% vs 13.0%, P = .02) and 24 months (0.5% vs 1.8%, P = .02) were lower with the AI algorithm than without it. Median radiologist interpretation time was lower with the AI algorithm than without it (102 seconds vs 142 seconds, P = .001). Conclusion Use of an artificial intelligence algorithm improved skeletal age assessment accuracy and reduced interpretation times for radiologists, although differences were observed between centers. Clinical trial registration no. NCT03530098 © RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Rubin in this issue.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287
External validity of randomised controlled trials: “To whom do the results of this trial apply?”
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(04)17670-8 · Cited by 2,544
Ting, D. S. W., Cheung, C. Y.-L., Lim, G., Tan, G. S. W., Quang, N. D., Gan, A., Hamzah, H., Garcia-Franco, R., San Yeo, I. Y., Lee, S. Y., et al. (2017) · JAMA · doi:10.1001/jama.2017.18152 · Cited by 2,402
Ardila, D., Kiraly, A. P., Bharadwaj, S., Choi, B., Reicher, J. J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0447-x · Cited by 2,167
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719
A Deep Learning Mammography-based Model for Improved Breast Cancer Risk Prediction
Yala, A., Lehman, C., Schuster, T., Portnoi, T., & Barzilay, R. (2019) · Radiology · doi:10.1148/radiol.2019182716 · Cited by 758
Diagnostic Accuracy of Digital Screening Mammography With and Without Computer-Aided Detection
Lehman, C. D., Wellman, R. D., Buist, D. S. M., Kerlikowske, K., Tosteson, A. N. A., & Miglioretti, D. L. (2015) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2015.5231 · Cited by 678
Kim, D. W., Jang, H. Y., Kim, K. W., Shin, Y., & Park, S. H. (2019) · Korean Journal of Radiology · doi:10.3348/kjr.2019.0025 · Cited by 459
Larson, D. B., Chen, M. C., Lungren, M. P., Halabi, S. S., Stence, N. V., & Langlotz, C. P. (2018) · Radiology · doi:10.1148/radiol.2017170236 · Cited by 456
Self-citationFully Automated Deep Learning System for Bone Age Assessment
Lee, H., Tajmir, S., Lee, J., Zissen, M., Yeshiwas, B. A., Alkasab, T. K., Choy, G., & Do, S. (2017) · Journal of Digital Imaging · doi:10.1007/s10278-017-9955-8 · Cited by 452
Cited by
AI recognition of patient race in medical imaging: a modelling study
Gichoya, J. W., Banerjee, I., Bhimireddy, A. R., Burns, J. L., Celi, L. A., Chen, L.-C., Correa, R., Dullerud, N., Ghassemi, M., Huang, S.-C., et al. (2022) · The Lancet Digital Health · doi:10.1016/s2589-7500(22)00063-2 · Cited by 611
“One study 11 found that an AI model could predict sex and distinguish between adult and paediatric patients from chest x-rays, while other studies 12 reported reasonable accuracy at predicting the chronological age of patients from various imaging studies.”
Self-citationRandomized Clinical Trials of Machine Learning Interventions in Health Care: A Systematic Review
Plana, D., Shung, D. L., Grimshaw, A. A., Saraf, A., Sung, J. J. Y., & Kann, B. H. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.33946 · Cited by 236
Methods for clinical evaluation of artificial intelligence algorithms for medical diagnosis
Park, S. H., Han, K., Jang, H. Y., Park, J. E., Lee, J.-G., Kim, D. W., & Choi, J. (2023) · Radiology · doi:10.1148/radiol.220182 · Cited by 163
McKee, M., & Wouters, O. J. (2022) · International Journal of Health Policy and Management · doi:10.34172/ijhpm.2022.7261 · Cited by 67
Wang, D.-Y., Ding, J., Sun, A.-L., Liu, S.-G., Jiang, D., Li, N., & Yu, J.-K. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad118 · Cited by 54
Metadata
- Authors
- Eng, D. K., Khandwala, N. B., Long, J., Fefferman, N. R., Lala, S. V., Strubel, N. A., Milla, S. S., Filice, R. W., Sharp, S. E., Towbin, A. J., Francavilla, M. L., Kaplan, S. L., Ecklund, K., Prabhu, S. P., Dillon, B. J., Everist, B. M., Anton, C. G., Bittman, M. E., Dennis, R., Larson, D. B., Seekins, J. M., Silva, C. T., Zandieh, A. R., Langlotz, C. P., Lungren, M. P., & Halabi, S. S.
- Venue
- Radiology
- Published
- 2021-12
- Volume
- 301
- Issue
- 3
- Pages
- 692-699
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W3204961660
- Cited by
- 84
- References
- 12
- FWCI
- 3.46
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Artificial Intelligence Algorithm Improves Radiologist Performance in Skeletal Age Assessment: A Prospective Multicenter Randomized Controlled Trial. https://app.cochat.ai/papers/10.1148/radiol.2021204021
- The paper
- Eng, D. K., Khandwala, N. B., Long, J., Fefferman, N. R., Lala, S. V., Strubel, N. A., Milla, S. S., Filice, R. W., Sharp, S. E., Towbin, A. J., Francavilla, M. L., Kaplan, S. L., Ecklund, K., Prabhu, S. P., Dillon, B. J., Everist, B. M., Anton, C. G., Bittman, M. E., Dennis, R., . . . Halabi, S. S. (2021). Artificial Intelligence Algorithm Improves Radiologist Performance in Skeletal Age Assessment: A Prospective Multicenter Randomized Controlled Trial. Radiology. https://doi.org/10.1148/radiol.2021204021
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.