CoChat
Sign Up
Cited by 62Artificial Intelligence in Healthcare and EducationRadiomics and Machine Learning in Medical ImagingMedical Image Segmentation Techniques

Image annotation and curation in radiology: an overview for machine learning practitioners

Open access
Galbusera, Fabio, Cina, Andrea·2024·European Radiology Experimental
DOI 10.1186/s41747-023-00408-yPMC PMC10844188
Open this paper in CoChat — free accountOpen PDF

Abstract

“Garbage in, garbage out” summarises well the importance of high-quality data in machine learning and artificial intelligence. All data used to train and validate models should indeed be consistent, standardised, traceable, correctly annotated, and de-identified, considering local regulations. This narrative review presents a summary of the techniques that are used to ensure that all these requirements are fulfilled, with special emphasis on radiological imaging and freely available software solutions that can be directly employed by the interested researcher. Topics discussed include key imaging concepts, such as image resolution and pixel depth; file formats for medical image data storage; free software solutions for medical image processing; anonymisation and pseudonymisation to protect patient privacy, including compliance with regulations such as the Regulation (EU) 2016/679 “General Data Protection Regulation” (GDPR) and the 1996 United States Act of Congress “Health Insurance Portability and Accountability Act” (HIPAA); methods to eliminate patient-identifying features within images, like facial structures; free and commercial tools for image annotation; and techniques for data harmonisation and normalisation. Relevance statement This review provides an overview of the methods and tools that can be used to ensure high-quality data for machine learning and artificial intelligence applications in radiology. Key points • High-quality datasets are essential for reliable artificial intelligence algorithms in medical imaging. • Software tools like ImageJ and 3D Slicer aid in processing medical images for AI research. • Anonymisation techniques protect patient privacy during dataset preparation. • Machine learning models can accelerate image annotation, enhancing efficiency and accuracy. • Data curation ensures dataset integrity, compliance, and quality for artificial intelligence development. Graphical Abstract

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s41747-023-00408-y

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Manual segmentation of medical images is labor-intensive, time-consuming, and subject to inter-observer variability, making it impractical for large-scale clinical applications.

    “However, manual segmentation of medical images is laborintensive, time-consuming, and subject to inter-observer variability, making it impractical for large-scale clinical applications [1, 2].”

    From Optimizing Neural Network Architecture for Medical Image Segmentation Using Monte Carlo Tree Search (2026) · cites this paper as [2]

    The Annotation section says that producing segmentation masks requires significant time and manual labour, especially for datasets containing thousands or millions of images. It does not address inter-observer variability or conclude that manual segmentation is impractical for large-scale clinical applications; its large-scale concern is annotating datasets for machine learning. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 35

  1. [1]Philbrick KA Weston AD Akkus Z RIL-Contour: a medical imaging dataset annotation tool for and with deep learning J Digit Imaging 2019 32 571 581 10.1007/s10278-019-00232-0 31089974 PMC6646456 · DOI
  2. [2]Hao Z Ge H Wang L Visual attention mechanism and support vector machine based automatic image annotation PLoS One 2018 13 11 e0206971 10.1371/journal.pone.0206971 30399159 PMC6219801 · DOI
  3. [3]Channin DS Mongkolwat P Kleper V Sepukar K Rubin DL The caBIG annotation and image markup project J Digit Imaging 2010 23 217 25 10.1007/s10278-009-9193-9 19294468 PMC2837161 · DOI
  4. [4]Monteiro E Costa C Oliveira JL A de-identification pipeline for ultrasound medical images in DICOM format J Med Syst 2017 41 89 10.1007/s10916-017-0736-1 28405948 · DOI
  5. [6]Swinburne NC Mendelson D Rubin DL Advancing semantic interoperability of image annotations: automated conversion of non-standard image annotations in a commercial PACS to the annotation and image markup J Digit Imaging 2020 33 49 53 10.1007/s10278-019-00191-6 30805778 PMC7064644 · DOI
  6. [7]Larobina M Murino L Medical image file formats J Digit Imaging 2014 27 200 206 10.1007/s10278-013-9657-9 24338090 PMC3948928 · DOI
  7. [8]Mildenberger P Eichelberg M Martin E Introduction to the DICOM standard Eur Radiol 2002 12 920 927 10.1007/s003300101100 11960249 · DOI
  8. [9]Kumar N Gupta R Gupta S Whole slide imaging (WSI) in pathology: current perspectives and future directions J Digit Imaging 2020 33 1034 1040 10.1007/s10278-020-00351-z 32468487 PMC7522141 · DOI
  9. [11]Li X Morgan PS Ashburner J Smith J Rorden C The first step for neuroimaging data analysis: DICOM to NIfTI conversion J Neurosci Methods 2016 264 47 56 10.1016/j.jneumeth.2016.03.001 26945974 · DOI · Paper page
  10. [12]Hanna MG Parwani A Sirintrapun SJ Whole slide imaging: technology and applications Adv Anat Pathol 2020 27 251 259 10.1097/PAP.0000000000000273 32452840 · DOI
  11. [13]Furht B, Akar E, Andrews WA (2018) Introduction to digital imaging. In: Digital image processing: practical approach. Briefs in Computer Science. Springer, Cham. 10.1007/978-3-319-96634-2_1 · DOI
  12. [15]Süsstrunk S, Buckley R, Swen S (1999) Standard RGB color spaces. Proc. IS&T/SID 7th Color Imaging Conference 7:127-134
  13. [16]Schneider CA Rasband WS Eliceiri KW NIH image to ImageJ: 25 years of image analysis Nat Methods 2012 9 671 675 10.1038/nmeth.2089 22930834 PMC5554542 · DOI · Paper page
  14. [17]Schindelin J Rueden CT Hiner MC Eliceiri KW The ImageJ ecosystem: an open platform for biomedical image analysis Mol Reprod Dev 2015 82 518 529 10.1002/mrd.22489 26153368 PMC5428984 · DOI
  15. [18]Schindelin J Arganda-Carreras I Frise E Fiji: an open-source platform for biological-image analysis Nat Methods 2012 9 676 682 10.1038/nmeth.2019 22743772 PMC3855844 · DOI · Paper page
  16. [19]Pieper S, Halle M, Kikinis R (2004) 3D Slicer. In: 2004 2nd IEEE International Symposium on Biomedical Imaging: Nano to Macro (IEEE Cat No. 04EX821). ieeexplore.ieee.org, pp 632–635 Vol. 1
  17. [20]Yushkevich PA Gao Yang Gerig G ITK-SNAP: an interactive tool for semi-automatic segmentation of multi-modality biomedical images Conf Proc IEEE Eng Med Biol Soc 2016 2016 3342 3345 PMC5493443 28269019
  18. [21]Freymann JB Kirby JS Perry JH Image data sharing for biomedical research—meeting HIPAA requirements for de-identification J Digit Imaging 2012 25 14 24 10.1007/s10278-011-9422-x 22038512 PMC3264712 · DOI
  19. [22]Diaz O Kushibar K Osuala R Data preparation for artificial intelligence in medical imaging: a comprehensive guide to open-access platforms and tools Phys Med 2021 83 25 37 10.1016/j.ejmp.2021.02.007 33684723 · DOI
  20. [23]Noumeir R Lemay A Lina J-M Pseudonymization of radiology data for research purposes J Digit Imaging 2007 20 284 295 10.1007/s10278-006-1051-4 17191099 PMC3043895 · DOI
  21. [25]Olatunji IE Rauch J Katzensteiner M Khosla M A review of anonymization for healthcare data Big Data 2022 10.1089/big.2021.0169 35271377 · DOI · Paper page
  22. [26]McClelland R Harper CM Information privacy in healthcare - the vital role of informed consent Eur J Health Law 2022 30 4 469 480 10.1163/15718093-bja10097 37582530 · DOI
  23. [27]Almalawi A Khan AI Alsolami F Abushark YB Alfakeeh AS Managing security of healthcare data for a modern healthcare system Sensors (Basel) 2023 23 3612 10.3390/s23073612 37050672 PMC10098823 · DOI
  24. [28]Coventry L Branley D Cybersecurity in healthcare: a narrative review of trends, threats and ways forward Maturitas 2018 113 48 52 10.1016/j.maturitas.2018.04.008 29903648 · DOI
  25. [31]Seh AH Zarour M Alenezi M Healthcare data breaches: insights and implications Healthcare (Basel) 2020 8 133 10.3390/healthcare8020133 32414183 PMC7349636 · DOI · Paper page
  26. [32]Fisher-Jeffes L Barton C Finlay F Clinicians’ knowledge of informed consent J Med Ethics 2007 33 181 4 10.1136/jme.2006.016758 17329394 PMC2598260 · DOI
  27. [33]Chadha NK Repanos C How much do healthcare professionals know about informed consent? A Bristol experience Surgeon 2004 2 328–33 360 10.1016/s1479-666x(04)80031-3 15712572 · DOI
  28. [34]Fetzer DT West OC The HIPAA privacy rule and protected health information: implications in research involving DICOM image databases Acad Radiol 2008 15 390 395 10.1016/j.acra.2007.11.008 18280936 · DOI
  29. [35]Castiglioni I Rundo L Codari M AI applications to medical images: from machine learning to deep learning Phys Med 2021 83 9 24 10.1016/j.ejmp.2021.02.006 33662856 · DOI
  30. [36]Dutta A Zisserman A The VIA annotation software for images, audio and video Proceedings of the 27th ACM International Conference on Multimedia 2019 New York Association for Computing Machinery
  31. [37]Lin T-Y Maire M Belongie S Microsoft COCO: common objects in context Computer vision – ECCV 2014 2014 Cham Springer International Publishing 740 755
  32. [39]Collewet G Strzelecki M Mariette F Influence of MRI acquisition protocols and image intensity normalization methods on texture classification Magn Reson Imaging 2004 22 81 91 10.1016/j.mri.2003.09.001 14972397 · DOI
  33. [40]Kociołek M Strzelecki M Obuchowicz R Does image normalization and intensity resolution impact texture classification? Comput Med Imaging Graph 2020 81 101716 10.1016/j.compmedimag.2020.101716 32222685 · DOI
  34. [41]Carré A Klausner G Edjlali M Standardization of brain MR images across machines and protocols: bridging the gap for MRI-based radiomics Sci Rep 2020 10 12340 10.1038/s41598-020-69298-z 32704007 PMC7378556 · DOI
  35. [42]Brahim A Jennane R Riad R A decision support tool for early detection of knee OsteoArthritis using X-ray imaging and machine learning: data from the OsteoArthritis Initiative Comput Med Imaging Graph 2019 73 11 18 10.1016/j.compmedimag.2019.01.007 30784984 · DOI

Cited by

Sorted by citations

Authors
Galbusera, F., & Cina, A.
Affiliations
  • Schulthess-Klinik (CH)
  • ETH Zurich (CH)
Corresponding author
Galbusera, Fabio
ORCID
Venue
European Radiology Experimental
Publisher
Springer Science and Business Media LLC
Published
2024-02-06
Volume
8
Issue
1
Pages
11-11
ISSN
2509-9280 (ISSN-L)
DOI
10.1186/s41747-023-00408-y
PMC
PMC10844188
OpenAlex
W4392734928
Cited by
62
References
2
Citations by year
2026: 26 · 2025: 27 · 2024: 9
FWCI
1.41
Type
journal article
Language
en
Open access
Licence
Locations
  • European Radiology Experimental (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Repository for Publications and Research Data (ETH Zurich) (submittedVersion, cc-by)
  • ETH Zürich Research Collection
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
Keywords
pseudonymization, patient privacy, GDPR, HIPAA, medical image processing software, ImageJ, 3D Slicer, data harmonization, data normalization, image resolution
MeSH terms
Machine Learning, Algorithms, Artificial Intelligence, Humans, Radiology, United States, Data Curation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review