Cited by 2,887Lung Cancer Diagnosis and TreatmentAI in cancer detectionRadiomics and Machine Learning in Medical Imaging

The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A Completed Reference Database of Lung Nodules on CT Scans

Open access
Armato, Samuel G., McLennan, Geoffrey, Bidaut, Luc, McNitt‐Gray, Michael F., Meyer, Charles R., Reeves, Anthony P., Zhao, Binsheng, Aberle, Denise R. +48 more·2011·Medical Physics

Abstract

Purpose: The development of computer‐aided diagnostic (CAD) methods for lung nodule detection, classification, and quantitative assessment can be facilitated through a well‐characterized repository of computed tomography (CT) scans. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI) completed such a database, establishing a publicly available reference for the medical imaging research community. Initiated by the National Cancer Institute (NCI), further advanced by the Foundation for the National Institutes of Health (FNIH), and accompanied by the Food and Drug Administration (FDA) through active participation, this public‐private partnership demonstrates the success of a consortium founded on a consensus‐based process. Methods: Seven academic centers and eight medical imaging companies collaborated to identify, address, and resolve challenging organizational, technical, and clinical issues to provide a solid foundation for a robust database. The LIDC/IDRI Database contains 1018 cases, each of which includes images from a clinical thoracic CT scan and an associated XML file that records the results of a two‐phase image annotation process performed by four experienced thoracic radiologists. In the initial blinded‐read phase, each radiologist independently reviewed each CT scan and marked lesions belonging to one of three categories (“ ,” “ ,” and “non‐ ”). In the subsequent unblinded‐read phase, each radiologist independently reviewed their own marks along with the anonymized marks of the three other radiologists to render a final opinion. The goal of this process was to identify as completely as possible all lung nodules in each CT scan without requiring forced consensus. Results: The Database contains 7371 lesions marked “nodule” by at least one radiologist. 2669 of these lesions were marked “ ” by at least one radiologist, of which 928 (34.7%) received such marks from all four radiologists. These 2669 lesions include nodule outlines and subjective nodule characteristic ratings. Conclusions: The LIDC/IDRI Database is expected to provide an essential medical imaging research resource to spur CAD development, validation, and dissemination in clinical practice.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1118/1.3528204

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • NoduleMNIST3D is a 3D dataset based on LIDC-IDRI, designed for binary classification of lung nodules by categorizing malignancy level 1/2 as negative and 4/5 as positive, ignoring level 3.

    “The NoduleMNIST3D is based on the LIDC-IDRI32, a large public lung nodule dataset, containing images from thoracic CT scans. The dataset is designed for both lung nodule segmentation and 5-level malignancy classification task. To perform binary classification, we categorize cases with malignancy level 1/2 into negative class and 4/5 into positive class, ignoring the cases with malignancy level 3.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 38

  1. R. M. Nishikawa, "Design of a common database for research in mam- mogram image analysis," Proc. SPIE 1905, 548-549 ͑1993͒. · DOI
  2. R. M. Nishikawa and L. M. Yarusso, "Variations in measured perfor- mance of CAD schemes due to database composition and scoring proto- col," Proc. SPIE 3338, 840-844 ͑1998͒. · DOI
  3. R. M. Nishikawa et al., "Effect of case selection on the performance of computer-aided detection schemes," Med. Phys. 21, 265-269 ͑1999͒. · DOI
  4. K. Bowyer et al., in "The digital database for screening mammography," Digital Mammography '96: Proceedings of the Third International Work- shop on Digital Mammography, edited by K. Doi, M. L. Giger, R. M. Nishikawa, and R. A. Schmidt ͑Elsevier, Amsterdam, 1996͒, pp. 431- 434.
  5. M. Heath et al., in "Current status of the digital database for screening mammography," Digital Mammography '98: Proceedings of the 4th In- ternational Workshop on Digital Mammography, edited by N. Karsse- meijer, M. Thijssen, J. Hendriks, and L. van Erning ͑Kluwer, Dordrecht, 1998͒, pp. 431-434.
  6. M. Heath, K. Bowyer, D. Kopans, R. Moore, and W. P. Kegelmeyer, in "The digital database for screening mammography," Digital Mammogra- phy 2000: Proceedings of the Fifth International Workshop on Digital Mammography, edited by M. J. Yaffe ͑Medical Physics, Madison, 2000͒, pp. 431-434.
  7. M. L. Giger, H.-P. Chan, and J. Boone, "Anniversary paper: History and status of CAD and quantitative image analysis: The role of Medical Phys- ics and AAPM," Med. Phys. 35, 5799-5820 ͑2008͒. · DOI · Paper page
  8. A. P. Reeves et al., "A public image database to support research in computer aided diagnosis," 31st Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2009, pp. 3715- 3718.
  9. S. G. Armato III et al., "The Reference Image Database to Evaluate Response to therapy in lung cancer ͑RIDER͒ project: A resource for the development of change analysis software," Clin. Pharmacol. Ther. 84, 448-456 ͑2008͒. · DOI
  10. K. W. Clark et al., "Creation of a CT image library for the Lung Screen- ing Study of the National Lung Screening Trial," J. Digit Imaging 20, 23-31 ͑2007͒. · DOI
  11. D. D. Cody et al., "Normalized CT dose index of the CT scanners used in the National Lung Screening Trial," AJR, Am. J. Roentgenol. 194, 1539- 1546 ͑2010͒. · DOI
  12. C. A. van Iersel et al., "Risk-based selection from the general population in a screening trial: Selection criteria, recruitment and power for the Dutch-Belgian randomised lung cancer multi-slice CT screening trial ͑NELSON͒," Int. J. Cancer 120, 868-874 ͑2007͒. · DOI
  13. D. M. Xu et al., "Nodule management protocol of the NELSON ran- domised lung cancer screening trial," Lung Cancer 54, 177-184 ͑2006͒. · DOI
  14. H. A. Gietema et al., "Pulmonary nodules detected at lung cancer screen- ing: Interobserver variability of semiautomated volume measurements," Radiology 241, 251-257 ͑2006͒. · DOI
  15. D. M. Xu et al., "Limited value of shape, margin and CT density in the discrimination between benign and malignant screen detected solid pul- monary nodules of the NELSON trial," Eur. J. Radiol. 68, 347-352 ͑2008͒. · DOI
  16. D. M. Xu et al., "Role of baseline nodule density and changes in density and nodule features in the discrimination between benign and malignant solid indeterminate pulmonary nodules," Eur. J. Radiol. 70, 492-498 ͑2009͒. · DOI
  17. K. Murphy, B. van Ginneken, A. M. R. Schilham, B. J. de Hoop, H. A. Gietema, and M. Prokop, "A large-scale evaluation of automatic pulmo- nary nodule detection in chest CT using local image features and k-nearest-neighbour classification," Med. Image Anal. 13, 757-770 ͑2009͒. · DOI
  18. B. van Ginneken et al., "Comparing and combining algorithms for computer-aided detection of pulmonary nodules in computed tomography scans: The ANODE09 study," Med. Image Anal. 14, 707-722 ͑2010͒. · DOI
  19. E. M. van Rikxoort, B. de Hoop, M. A. Viergever, M. Prokop, and B. van Ginneken, "Automatic lung segmentation from thoracic computed tomog- raphy scans using a hybrid approach with error detection," Med. Phys. 36, 2934-2947 ͑2009͒. · DOI
  20. T. Doshi, D. Rusinak, R. A. Halvorsen, D. C. Rockey, K. Suzuki, and A. H. Dachman, "CT colonography: False-negative interpretations," Radiol- ogy 244, 165-173 ͑2007͒. · DOI
  21. L. P. Clarke, B. Y. Croft, E. Staab, H. Baker, and D. C. Sullivan, "Na- tional Cancer Institute initiative: Lung image database resource for imag- ing research," Acad. Radiol. 8, 447-450 ͑2001͒. · DOI
  22. L. E. Dodd et al., "Assessment methodologies and statistical issues for computer-aided diagnosis of lung nodules in computed tomography: Con- temporary research topics relevant to the Lung Image Database Consor- tium," Acad. Radiol. 11, 462-475 ͑2004͒. · DOI
  23. S. G. Armato III et al., "Lung Image Database Consortium: Developing a resource for the medical imaging research community," Radiology 232, 739-748 ͑2004͒. · DOI
  24. M. C. Carrillo, C. A. Sanders, and R. G. Katz, "Maximizing the Alzhe- imer's Disease Neuroimaging Initiative II," Alzheimers and Dementia 5, 271-275 ͑2009͒. · DOI
  25. M. F. McNitt-Gray et al., "The Lung Image Database Consortium ͑LIDC͒ data collection process for nodule detection and annotation," Acad. Ra- diol. 14, 1464-1474 ͑2007͒. · DOI
  26. S. G. Armato III et al., "The Lung Image Database Consortium ͑LIDC͒: An evaluation of radiologist variability in the identification of lung nod- ules on CT scans," Acad. Radiol. 14, 1409-1421 ͑2007͒. · DOI
  27. C. R. Meyer et al., "Evaluation of lung MDCT nodule annotations across radiologists and methods," Acad. Radiol. 13, 1254-1265 ͑2006͒. · DOI
  28. I. Petkovska et al., "The effect of lung volume on nodule size on CT," Acad. Radiol. 14, 476-485 ͑2007͒. · DOI
  29. N. Otsu, "A threshold selection method from grey level histograms," IEEE Trans. Syst. Man Cybern. SMC-9, 62-66 ͑1979͒.
  30. S. G. Armato III et al., "The Lung Image Database Consortium ͑LIDC͒: Ensuring the integrity of expert-defined "truth"," Acad. Radiol. 14, 1455- 1463 ͑2007͒. · DOI
  31. A. P. Reeves et al., "The Lung Image Database Consortium ͑LIDC͒: A comparison of different size metrics for pulmonary nodule measure- ments," Acad. Radiol. 14, 1475-1485 ͑2007͒. · DOI
  32. R. Ochs, H. J. Kim, E. Angel, C. Panknin, M. McNitt-Gray, and M. Brown, "Forming a reference standard from LIDC data: Impact of reader agreement on reported CAD performance," Proc. SPIE 6514, 65142A-1- 65142A-6 ͑2007͒. · DOI
  33. W. F. Sensakovic, A. Starkey, R. Y. Roberts, and S. G. Armato III, "Dis- crete space vs. continuous space lesion boundary and area definition," Med. Phys. 35, 4070-4078 ͑2008͒. · DOI
  34. A. M. Biancardi, A. C. Jirapatnakul, and A. P. Reeves, "A comparison of ground truth estimation methods," International Journal of Computer As- sisted Radiology and Surgery 5, 295-305 ͑2010͒. · DOI
  35. W. H. Horsthemke, D. S. Raicu, and J. D. Furst, "Evaluation challenges for bridging semantic gap: Shape disagreements on pulmonary nodules in the Lung Image Database Consortium," International Journal of Health- care Information Systems and Informatics 4, 17-33 ͑2008͒.
  36. D. Zinovev, D. S. Raicu, J. D. Furst, and S. G. Armato III, "Predicting radiological panel opinions using a panel of machine learning classifiers," J. Algorithms 2, 1473-1502 ͑2009͒. · DOI
  37. S. G. Armato III et al., "Assessment of radiologist performance in the detection of lung nodules: Dependence on the definition of "truth"," Acad. Radiol. 16, 28-38 ͑2009͒. · DOI
  38. A. J. Buckler et al., "Volumetric CT in lung cancer: An example for the qualification of imaging as a biomarker," Acad. Radiol. 17, 107-115 ͑2010͒. · DOI

Cited by

Sorted by citations

Authors
Armato, S. G., McLennan, G., Bidaut, L., McNitt‐Gray, M. F., Meyer, C. R., Reeves, A. P., Zhao, B., Aberle, D. R., Henschke, C. I., Hoffman, E. A., Kazerooni, E. A., MacMahon, H., van Beek, E. J. R., Yankelevitz, D., Biancardi, A. M., Bland, P. H., Brown, M. S., Engelmann, R. M., Laderach, G. E., Max, D., Pais, R. C., Qing, D. P. ‐., Roberts, R. Y., Smith, A. R., Starkey, A., Batra, P., Caligiuri, P., Farooqi, A., Gladish, G. W., Jude, C. M., Munden, R. F., Petkovska, I., Quint, L. E., Schwartz, L. H., Sundaram, B., Dodd, L. E., Fenimore, C., Gur, D., Petrick, N., Freymann, J., Kirby, J., Hughes, B., Vande Casteele, A., Gupte, S., Sallam, M., Heath, M. D., Kuhn, M. H., Dharaiya, E., Burns, R., Fryd, D. S., Salganicoff, M., Anand, V., Shreter, U., Vastagh, S., Croft, B. Y., & Clarke, L. P.
Affiliations
  • University of Chicago (US)
  • University of Iowa (US)
  • The University of Texas MD Anderson Cancer Center (US)
  • University of California, Los Angeles (US)
  • University of Michigan (US)
  • Cornell University (US)
  • Memorial Sloan Kettering Cancer Center (US)
  • Weill Cornell Medicine (US)
  • Michigan Medicine (US)
  • Louisiana State University (US)
  • University of Utah (US)
  • Beth Israel Deaconess Medical Center (US)
  • National Institute of Allergy and Infectious Diseases (US)
  • National Institute of Standards and Technology (US)
  • University of Pittsburgh (US)
  • United States Food and Drug Administration (US)
  • Science Applications International Corporation (United States) (US)
  • Fujifilm (United States) (US)
  • Carestream (United States) (US)
  • Philips (Germany) (DE)
  • Siemens Healthcare (United States) (US)
  • National Cancer Institute (US)
Corresponding author
Armato, Samuel G.
Venue
Medical Physics
Publisher
Wiley
Published
2011-01-24
Volume
38
Issue
2
Pages
915-931
ISSN
  • 0094-2405 (ISSN-L)
  • 2473-4209
OpenAlex
W1986649315
Cited by
2,887
References
0
Citations by year
2026: 320 · 2025: 350 · 2024: 320 · 2023: 339 · 2022: 319 · 2021: 281 · 2020: 257 · 2019: 237 · 2018: 162 · 2017: 106 · 2016: 78 · 2015: 49 · 2014: 27 · 2013: 27 · 2012: 8
FWCI
18.23
Type
journal article
Language
en
Open access
Locations
  • Medical Physics (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • Discovery Research Portal (University of Dundee) (publishedVersion)
  • Deep Blue (University of Michigan) (submittedVersion)
  • Figshare (submittedVersion, cc-by)
Funders
  • U.S. Public Health Service (U01CA091085, U01CA091090, U01CA091099, U01CA091100, U01CA091103)
  • National Cancer Institute (HHSN261200800001E)
Topics
  • Health Sciences › Medicine › Pulmonary and Respiratory Medicine › Lung Cancer Diagnosis and Treatment
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
Keywords
pulmonary nodules, computed tomography, reference database, computer-aided diagnosis, nodule detection, nodule classification, radiologist annotation, chest CT, public-private partnerships
MeSH terms
Diagnosis, Computer-Assisted, Humans, Lung, Lung Neoplasms, Quality Control, Radiographic Image Interpretation, Computer-Assisted, Reference Standards, Radiography, Thoracic, Tomography, X-Ray Computed, Databases, Factual, Tumor Burden

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review