Principles for evaluating the clinical implementation of novel digital healthcare devices
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
With growing interest in novel digital healthcare devices, such as artificial intelligence (AI) software for medical diagnosis and prediction, and their potential impacts on healthcare, discussions have taken place regarding the regulatory approval, coverage, and clinical implementation of these devices. Despite their potential, 'digital exceptionalism' (i.e., skipping the rigorous clinical validation of such digital tools) is creating significant concerns for patients and healthcare stakeholders. This white paper presents the positions of the Korean Society of Radiology, a leader in medical imaging and digital medicine, on the clinical validation, regulatory approval, coverage decisions, and clinical implementation of novel digital healthcare devices, especially AI software for medical diagnosis and prediction, and explains the scientific principles underlying those positions. Mere regulatory approval by the Food and Drug Administration of Korea, the United States, or other countries should be distinguished from coverage decisions and widespread clinical implementation, as regulatory approval only indicates that a digital tool is allowed for use in patients, not that the device is beneficial or recommended for patient care. Coverage or widespread clinical adoption of AI software tools should require a thorough clinical validation of safety, high accuracy proven by robust external validation, documented benefits for patient outcomes, and cost-effectiveness. The Korean Society of Radiology puts patients first when considering novel digital healthcare tools, and as an impartial professional organization that follows scientific principles and evidence, strives to provide correct information to the public, make reasonable policy suggestions, and build collaborative partnerships with industry and government for the good of our patients.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Convolutional neural networks: an overview and application in radiology
Yamashita, R., Nishio, M., Do, R. K. G., & Togashi, K. (2018) · Insights into Imaging · doi:10.1007/s13244-018-0639-9 · Cited by 4,682 · cited 2×
“이는 이러한 복잡한 기계학습 소프트 웨어 알고리듬은 학습에 사용된 자료 내에서는 소프트웨어 의 진단/예측 정확도가 매우 높은 반면 외부 자료로 일반화 가 어려울 수 있다는 특징이 있기 때문이다[10,11,18,2126].”
Screening for Colorectal Cancer: US Preventive Services Task Force Recommendation Statement
US Preventive Services Task Force, Bibbins-Domingo, K., Grossman, D. C., Curry, S. J., Davidson, K. W., Epling, J. W., García, F. A. R., Gillman, M. W., Harper, D. M., Kemper, A. R., et al. (2016) · JAMA · doi:10.1001/jama.2016.5989 · Cited by 2,509
Ting, D. S. W., Cheung, C. Y.-L., Lim, G., Tan, G. S. W., Quang, N. D., Gan, A., Hamzah, H., Garcia-Franco, R., San Yeo, I. Y., Lee, S. Y., et al. (2017) · JAMA · doi:10.1001/jama.2017.18152 · Cited by 2,402
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719 · cited 1×
“예를 들어, 의료영상을 분석하는 소프트웨어 알고리듬 의 경우 학습에 이용되지 않은 다른 병원의 자료나 다른 영 상장비로 얻은 자료에 적용할 경우 정확도가 낮아진다[26].”
Deep Learning: A Primer for Radiologists
Chartrand, G., Cheng, P. M., Vorontsov, E., Drozdzal, M., Turcotte, S., Pal, C. J., Kadoury, S., & Tang, A. (2017) · RadioGraphics · doi:10.1148/rg.2017170077 · Cited by 1,115 · cited 2×
“인공지능은 인간의 인지능력이 필요한 행위를 컴퓨터가 수행할 수 있도록 하는 시스템 혹은 알고리듬을 통칭하는 넓은 의미의 용어로서, 그 중에서 기계학습은 행위 알고리 듬의 명시적인 프로그래밍 없이 기계가 데이터를 기반으로 자료 내의 어떤 패턴을 찾아(이 과정을 학습이라 부른다) 인 공지능을 수행할 수 있도록 하는 것을 지칭한다(Figure 1) [9,10].”
Luo, W., Phung, D., Tran, T., Gupta, S., Rana, S., Karmakar, C., Shilton, A., Yearwood, J., Dimitrova, N., Ho, T. B., et al. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.5870 · Cited by 988
Park, S. H., & Han, K. (2018) · Radiology · doi:10.1148/radiol.2017171920 · Cited by 894
Self-citationCurrent Applications and Future Impact of Machine Learning in Radiology
Choy, G., Khalilzadeh, O., Michalski, M., Do, S., Samir, A. E., Pianykh, O. S., Geis, J. R., Pandharipande, P. V., Brink, J. A., & Dreyer, K. J. (2018) · Radiology · doi:10.1148/radiol.2018171820 · Cited by 800
Diagnostic Accuracy of Digital Screening Mammography With and Without Computer-Aided Detection
Lehman, C. D., Wellman, R. D., Buist, D. S. M., Kerlikowske, K., Tosteson, A. N. A., & Miglioretti, D. L. (2015) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2015.5231 · Cited by 678
Steinhubl, S. R., Waalen, J., Edwards, A. M., Ariniello, L. M., Mehta, R. R., Ebner, G. S., Carter, C., Baca-Motes, K., Felicione, E., Sarich, T., et al. (2018) · JAMA · doi:10.1001/jama.2018.8102 · Cited by 539 · cited 7×
“또한, 2017년에는 싱 가폴 의료진이 주축이 되어 수행한 다국적 다기관 연구가 JAMA에 발표되었는데[36], 이 연구는 7만 건이 넘는 영상 으로 학습한 딥러닝기반 인공지능 소프트웨어를 중국, 싱가 폴, 미국, 오스트레일리아, 멕시코, 홍콩에 있는 10개의 다 양한 상황의 외부 의료기관의 자료를 이용하여 진단능을 검 증하였고 인공지능 소프트웨어가 당뇨망막병증 진단에 있 어 높은 정확도를 낼 수 있음을 재차 확인하였다.”
Influential
Cited by
Kim, D. W., Jang, H. Y., Kim, K. W., Shin, Y., & Park, S. H. (2019) · Korean Journal of Radiology · doi:10.3348/kjr.2019.0025 · Cited by 459
“For example, computer scientists typically evaluate the performance of AI algorithms on “test” datasets; however, these are usually random subsamples of the original dataset, and thus, adequate external validation of clinical performance is not possible (10, 16, 20, 25).”
BackgroundMethodSelf-citationUnderstanding artificial intelligence based radiology studies: What is overfitting?
Mutasa, S., Sun, S., & Ha, R. (2020) · Clinical Imaging · doi:10.1016/j.clinimag.2020.04.025 · Cited by 216
What should medical students know about artificial intelligence in medicine?
Park, S. H., Do, K.-H., Kim, S., Park, J. H., & Lim, Y.-S. (2019) · Journal of Educational Evaluation for Health Professions · doi:10.3352/jeehp.2019.16.18 · Cited by 196
Self-citationvan de Sande, D., Van Genderen, M. E., Smit, J. M., Huiskens, J., Visser, J. J., Veen, R. E. R., van Unen, E., BA, O. H., Gommers, D., & Bommel, J. V. (2022) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100495 · Cited by 135
Artificial intelligence for ultrasonography: unique opportunities and challenges
Park, S. H. (2021) · Ultrasonography · doi:10.14366/usg.20078 · Cited by 37
Self-citation
Metadata
- Authors
- Park, S. H., Do, K.-H., Choi, J.-I., Sim, J. S., Yang, D. M., Eo, H., Woo, H., Lee, J. M., Jung, S. E., & Oh, J. H.
- Venue
- Journal of the Korean Medical Association
- Publisher
- Korean Medical Association (KAMJE)
- Published
- 2018
- Volume
- 61
- Issue
- 12
- Pages
- 765
- ISSN
- 1975-8456 (ISSN-L)
- 2093-5951
- OpenAlex
- W2904583321
- Cited by
- 18
- References
- 15
- FWCI
- 0.65
- Type
- journal article
- Language
- ko
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Principles for evaluating the clinical implementation of novel digital healthcare devices. https://app.cochat.ai/papers/10.5124/jkma.2018.61.12.765
- The paper
- Park, S. H., Do, K.-H., Choi, J.-I., Sim, J. S., Yang, D. M., Eo, H., Woo, H., Lee, J. M., Jung, S. E., & Oh, J. H. (2018). Principles for evaluating the clinical implementation of novel digital healthcare devices. Journal of the Korean Medical Association. https://doi.org/10.5124/jkma.2018.61.12.765
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.