CoChat
Sign Up
Cited by 22Visual perception and processing mechanismsVisual Attention and Saliency DetectionRadiology practices and education

Under-exploration of three-dimensional images leads to search errors for small salient targets

Open access
Lago, Miguel A., Jonnalagadda, Aditya, Abbey, Craig K., Barufaldi, Bruno B., Bakic, Predrag R., Maidment, Andrew D.A., Leung, Winifred K., Weinstein, Susan P. +2 more·2021·Current Biology
DOI 10.1016/j.cub.2020.12.029PMC PMC8048135
Open this paper in CoChat — free accountOpen PDF

Abstract

Advances in 3D imaging technology are transforming how radiologists search for cancer 1,2 and how security officers scrutinize baggage for dangerous objects.3 These new 3D technologies often improve search over 2D images 4,5 but vastly increase the image data. Here, we investigate 3D search for targets of various sizes in filtered noise and digital breast phantoms. For a Bayesian ideal observer optimally processing the filtered noise and a convolutional neural network processing the digital breast phantoms, search with 3D image stacks increases target information and improves accuracy over search with 2D images. In contrast, 3D search by humans leads to high miss rates for small targets easily detected in 2D search, but not for larger targets more visible in the visual periphery. Analyses of human eye movements, perceptual judgments, and a computational model with a foveated visual system suggest that human errors can be explained by interaction among a target's peripheral visibility, eye movement under-exploration of the 3D images, and a perceived overestimation of the explored area. Instructing observers to extend the search reduces 75% of the small target misses without increasing false positives. Results with twelve radiologists confirm that even medical professionals reading realistic breast phantoms have high miss rates for small targets in 3D search. Thus, under-exploration represents a fundamental limitation to the efficacy with which humans search in 3D image stacks and miss targets with these prevalent image technologies.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.cub.2020.12.029

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Complex objects can only be identified within the useful field of view, roughly 2.5° centered on the fovea.

    “Complex objects can only be identified within the useful field of view , which is roughly 2.5 ° centered on the fovea [9 , 19] , equivalent to a 4.4 cm width when looking at a screen 1 m away.”

    From Enhancing human-AI collaboration: The case of colonoscopy (2024) · cites this paper as [19]

    The Results and Discussion section defines the useful field of view as a 2.5°-radius circle around each fixation. It reports that small-target detectability declines with retinal eccentricity, but does not establish that complex objects can only be identified inside that circle; it also reports that larger targets remain more detectable in the periphery. No page number is available in our copy. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • Detection ability degrades on the periphery of the visual field because of reduced visual acuity and crowding.

    “In addition, without us noticing, detection ability degrades on the periphery of the visual field ( Fig. 1 ) because of reduced visual acuity and crowding [19 , 20] .”

    From Enhancing human-AI collaboration: The case of colonoscopy (2024) · cites this paper as [19]

    The Results and Discussion (page unavailable) report that small-target detectability drops sharply with retinal eccentricity. The Methods (page unavailable) describe a model that loses access to high spatial frequencies farther from fixation, consistent with reduced acuity. Our copy does not identify crowding as a cause of the decline, so it does not support the full explanation attributed to this source. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Human memory for already sampled locations is poor, resulting in less efficient and defective search behavior.

    “However, human memory for already sampled locations proved to be poor, resulting in both less efficient and defective search behavior: less efficient because they may look multiple times at the same location, defective because they may miss a significant proportion of the image [19 , 21 , 22] .”

    From Enhancing human-AI collaboration: The case of colonoscopy (2024) · cites this paper as [19]

    In Results and Discussion (page not available), observers vastly overestimated how much of 3D image stacks they had explored, and extending their search reduced misses for small targets. This supports the claim that incomplete exploration can cause search errors, but the paper does not establish poor memory for already sampled locations or repeated looks at the same location as the cause of inefficiency. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 70

  1. [1]Rogalla P, Kloeters C, and Hein PA (2009). CT technology overview: 64-slice and beyond. Radiol. Clin. North Am 47, 1– 11. 19195530
  2. [2]Skaane P. (2017). Breast cancer screening with digital breast tomosynthesis. Breast Cancer 24, 32– 41. 27138386
  3. [4]Georgian-Smith D, Obuchowski NA, Lo JY, Brem RF, Baker JA, Fisher PR, Rim A, Zhao W, Fajardo LL, and Mertelmeier T. (2019). Can digital breast tomosynthesis replace full-field digital mammography? A multireader, multicase study of wide-angle tomosynthesis. Am. J. Roentgenol 212, 1393– 1399. 30933648
  4. [5]Mayo JR, and Lam S. (2015). Computed tomography and the secrets of lung nodules. Can. Assoc. Radiol. J 66, 2– 4. 25623006
  5. [6]Wolfe JM, and Horowitz TS (2017). Five factors that guide attention in visual search. Nat. Hum. Behav 1, 0058. PMC9879335 36711068
  6. [7]Eckstein MP, Koehler K, Welbourne LE, and Akbas E. (2017). Humans, but not deep neural networks, often miss giant targets in scenes. Curr. Biol 27, 2827– 2832.e3. 28889976
  7. [8]Malcolm GL, and Henderson JM (2009). The effects of target template specificity on visual search in real-world scenes: evidence from eye movements. J. Vis 9, 8. 20053071
  8. [9]Eckstein MP (2017). Probabilistic computations for attention, eye movements, and search. Annu. Rev. Vis. Sci 3, 319– 342. 28746814
  9. [10]Peelen MV, and Kastner S. (2014). Attention in the real world: toward understanding its neural basis. Trends Cogn. Sci 18, 242– 250. 24630872 PMC4908952
  10. [11]Wolfe JM, Võ ML-H, Evans KK, and Greene MR (2011). Visual search in scenes involves selective and nonselective pathways. Trends Cogn. Sci 15, 77– 84. 21227734 PMC3035167
  11. [12]Zhang M, Feng J, Ma KT, Lim JH, Zhao Q, and Kreiman G. (2018). Finding any Waldo with zero-shot invariant and efficient visual search. Nat. Commun 9, 3730. 30213937 PMC6137219
  12. [13]Verghese P. (2012). Active search for multiple targets is inefficient. Vision Res. 74, 61– 71. 22929812
  13. [14]Wolfe JM, Horowitz TS, and Kenner NM (2005). Cognitive psychology: rare items often missed in visual searches. Nature 435, 439– 440. 15917795 PMC4224304
  14. [15]Michel M, and Geisler WS (2011). Intrinsic position uncertainty explains detection and localization performance in peripheral vision. J. Vis 11, 18. PMC3243920 21257707
  15. [16]Paulun VC, Schütz AC, Michel MM, Geisler WS, and Gegenfurtner KR (2015). Visual search under scotopic lighting conditions. Vision Res. 113 (Pt B), 155– 168. 25988753 PMC5645078
  16. [17]Mitroff SR, and Biggs AT (2014). The ultra-rare-item effect: visual search for exceedingly rare items is highly susceptible to error. Psychol. Sci 25, 284– 289. 24270463
  17. [18]Semizer Y, and Michel MM (2017). Intrinsic position uncertainty impairs overt search performance. J. Vis 17, 13. 28837969
  18. [19]Ackermann JF, and Landy MS (2010). Suboptimal choice of saccade endpoint in search with unequal payoffs. J. Vis 10, 530. PMC4172012 24023277
  19. [20]Li Z, Desolneux A, Muller S, de Carvalho PM, and Carton A-K(2018). Comparison of microcalcification detectability in FFDM and DBT using a virtual clinical trial. Proc. SPIE 10577, 105770D.
  20. [21]Gur D, Abrams GS, Chough DM, Ganott MA, Hakim CM, Perrin RL, Rathfon GY, Sumkin JH, Zuley ML, and Bandos AI (2009). Digital breast tomosynthesis: observer performance study. AJR Am. J. Roentgenol 193, 586– 591. 19620460
  21. [23]Williams LH, and Drew T. (2019). What do we know about volumetric medical image interpretation?: a review of the basic science and medical image perception literatures. Cogn. Res. Princ. Implic 4, 21. 31286283 PMC6614227
  22. [24]Wu C-C, and Wolfe JM (2019). Eye movements in medical image perception: a selective review of past, present and future. Vision (Basel) 3, 32. PMC6802791 31735833
  23. [25]Karimi S, Jiang X, Cosman P, and Martz H. (2014). Flexible methods for segmentation evaluation: results from CT-based luggage screening. J. XRay Sci. Technol 22, 175– 195. 24699346 PMC5443714
  24. [26]Ferrand G, English J, and Irani P. (2016). 3D visualization of astronomy data cubes using immersive displays. arXiv, arXiv:1607.08874v1. https://arxiv.org/abs/1607.08874. · arXiv
  25. [28]Drew T, Vo ML, Olwal A, Jacobson F, Seltzer SE, and Wolfe JM(2013). Scanners and drillers: characterizing expert visual search through volumetric images. J. Vis 13, 3. PMC3736761 23922445
  26. [29]Aizenman A, Drew T, Ehinger KA, Georgian-Smith D, and Wolfe JM(2017). Comparing search patterns in digital breast tomosynthesis and full-field digital mammography: an eye tracking study. J. Med. Imaging (Bellingham) 4, 045501. 29098168 PMC5658515
  27. [30]Noroozian M, Hadjiiski L, Rahnama-Moghadam S, Klein KA, Jeffries DO, Pinsky RW, Chan HP, Carson PL, Helvie MA, and Roubidoux MA (2012). Digital breast tomosynthesis is comparable to mammographic spot views for mass characterization. Radiology 262, 61– 68. 21998048 PMC3244671
  28. [31]Cohn TE, and Lasley DJ (1974). Detectability of a luminance increment: effect of spatial uncertainty. J. Opt. Soc. Am 64, 1715– 1719. 4443844
  29. [32]Palmer J. (1994). Set-size effects in visual search: the effect of attention is independent of the stimulus for simple tasks. Vision Res. 34, 1703– 1721. 7941377
  30. [33]Burgess AE, and Ghandeharian H. (1984). Visual signal detection. II. Signal-location identification. J. Opt. Soc. Am. A 1, 906– 910. 6470843
  31. [34]Bochud FO, Abbey CK, and Eckstein MP (2004). Search for lesions in mammograms: statistical characterization of observer responses. Med. Phys 31, 24– 36. 14761017
  32. [35]Burgess AE, Wagner RF, Jennings RJ, and Barlow HB (1981). Efficiency of human visual signal discrimination. Science 214, 93– 94. 7280685
  33. [36]Geisler WS (2011). Contributions of ideal observer theory to vision research. Vision Res. 51, 771– 781. 20920517 PMC3062724
  34. [37]Barlow HB (1980). The absolute efficiency of perceptual decisions. Philos. Trans. R. Soc. Lond. B Biol. Sci 290, 71– 82. 6106243
  35. [38]Eckstein MP, Whiting JS, and Thomas JP (1996). Role of knowledge in human visual temporal integration in spatiotemporal noise. J. Opt. Soc. Am. A Opt. Image Sci. Vis 13, 1960– 1968. 8828198
  36. [39]Kundel HL (1975). Peripheral vision, structured noise and film reader error. Radiology 114, 269– 273. 1110990
  37. [40]Kundel HL, Nodine CF, Thickman D, and Toto L. (1987). Searching for lung nodules. A comparison of human performance with random and systematic scanning models. Invest. Radiol 22, 417– 422. 3597010
  38. [41]Krupinski EA (2011). The role of perception in imaging: past and future. Semin. Nucl. Med 41, 392– 400. 21978443
  39. [42]Eckstein MP, Beutter BR, and Stone LS (2001). Quantifying the performance limits of human saccadic targeting during visual search. Perception 30, 1389– 1401. 11768491
  40. [43]Najemnik J, and Geisler WS (2008). Eye movement statistics in humans are consistent with an optimal search strategy. J. Vis 8, 4. PMC2868380 18484810
  41. [44]Ludwig CJH, Davies JR, and Eckstein MP (2014). Foveal analysis and peripheral selection during active visual sampling. Proc. Natl. Acad. Sci. USA 111, E291– E299. 24385588 PMC3896144
  42. [45]Akbas E, and Eckstein MP (2017). Object detection through search with a foveated visual system. PLoS Comput. Biol 13, e1005743. 28991906 PMC5669499
  43. [46]Myers KJ, and Barrett HH (1987). Addition of a channel mechanism to the ideal-observer model. J. Opt. Soc. Am. A 4, 2447– 2457. 3430229
  44. [47]Shimozaki SS, Eckstein MP, and Abbey CK (2003). An ideal observer with channels versus feature-independent processing of spatial frequency and orientation in visual search performance. J. Opt. Soc. Am. A Opt. Image Sci. Vis 20, 2197– 2215. 14686499
  45. [48]Araujo C, Kowler E, and Pavel M. (2001). Eye movements during visual search: the costs of choosing the optimal path. Vision Res. 41, 3613– 3625. 11718799
  46. [49]Kowler E. (2011). Eye movements: the past 25 years. Vision Res. 51, 1457– 1483. 21237189 PMC3094591
  47. [50]Pokrajac DD, Maidment AD, and Bakic PR (2012). Optimized generation of high resolution breast anthropomorphic software phantoms. Med. Phys 39, 2290– 2302. 22482649 PMC3337667
  48. [51]Bakic PR, Pokrajac DD, De Caro R, and Maidment ADA (2014). Realistic simulation of breast tissue microstructure in software anthropomorphic phantoms. In Breast Imaging: 12th International Workshop, IWDM 2014, Gifu City, Japan, June 29–July 2, 2014. Proceedings, Fujita H, Hara T, and Muramatsu C, eds. (Springer International), pp. 348– 355.
  49. [52]Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, and Ronneberger O. (2016). 3D U-Net: learning dense volumetric segmentation from sparse annotation. arXiv, arXiv:1606.06650v1. https://arxiv.org/abs/1606.06650v1. · arXiv
  50. [53]Zuckerman SP, Maidment ADA, Weinstein SP, McDonald ES, and Conant EF (2017). Imaging with synthesized 2D mammography: differences, advantages, and pitfalls compared with digital mammography. AJR Am. J. Roentgenol 209, 222– 229. 28463546
  51. [54]Zuckerman SP, Conant EF, Keller BM, Maidment ADA, Barufaldi B, Weinstein SP, Synnestvedt M, and McDonald ES (2016). Implementation of synthesized two-dimensional mammography in a population-based digital breast tomosynthesis screening program. Radiology 281, 730– 736. 27467468 PMC5131829
  52. [55]Greenspan H, van Ginneken B, and Summers RM (2016). Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans. Med. Imaging 35, 1153– 1159.
  53. [56]Rodriguez-Ruiz A, Lång K, Gubern-Merida A, Teuwen J, Broeders M, Gennaro G, Clauser P, Helbich TH, Chevalier M, Mertelmeier T, (2019). Can we reduce the workload of mammographic screening by automatic identification of normal exams with artificial intelligence? A feasibility study. Eur. Radiol 29, 4825– 4832. 30993432 PMC6682851
  54. [57]McKinney SM, Sieniek M, Godbole V, Godwin J, Antropova N, Ashrafian H, Back T, Chesus M, Corrado GS, Darzi A, (2020). International evaluation of an AI system for breast cancer screening. Nature 577, 89– 94. 31894144
  55. [58]Juni MZ, and Eckstein MP (2017). The wisdom of crowds for visual search. Proc. Natl. Acad. Sci. USA 114, E4306– E4315. 28490500 PMC5448189
  56. [60]Caumo F, Brunelli S, Zorzi M, Baglio I, Ciatto S, and Montemezzi S. (2011). Benefits of double reading of screening mammograms: retrospective study on a consecutive series. Radiol. Med. (Torino) 116, 575– 583. 21424314
  57. [62]Peirce JW (2007). PsychoPy–psychophysics software in Python. J. Neurosci. Methods 162, 8– 13. 17254636 PMC2018741
  58. [63]Isensee F, Petersen J, Kohl SA, Jäger PF, and Maier-Hein KH (2019). nnu-net: breaking the spell on successful medical image segmentation. arXiv, arXiv:1904.08128v1. https://arxiv.org/abs/1904.08128v1. · arXiv
  59. [64]Bakic PR, Pokrajac DD, and Maidment ADA (2017). Computer simulation of the breast subcutaneous and retromammary tissue for use in virtual clinical trials. Proc. SPIE 10132, 101325C.
  60. [65]Bakic PR, Barufaldi B, Higginbotham D, Weinstein SP, Avanaki AN, Espig KS, Xthona A, Kimpe TRL, and Maidment ADA (2018). Virtual clinical trial of lesion detection in digital mammography and digital breast tomosynthesis. Proc. SPIE 10573, 1057306.
  61. [67]Geisler WS (2003). Ideal Observer Analysis (MIT). 2652171
  62. [68]Burgess A, and Ghandeharian H. (1984). Visual signal detection. I. Ability to use phase information. J. Opt. Soc. Am. A 1, 900– 905. 6470842
  63. [69]Abbey CK, and Eckstein MP (2014). Observer efficiency in free-localization tasks with correlated noise. Front. Psychol 5, 345. 24817854 PMC4013476
  64. [70]Peterson W, Birdsall T, and Fox W. (1954). The theory of signal detectability. Trans. IRE Profess. Group Info. Theory 4, 171– 212.
  65. [71]Yu L, Chen B, Kofler JM, Favazza CP, Leng S, Kupinski MA, and McCollough CH (2017). Correlation between a 2D channelized Hotelling observer and human observers in a low-contrast detection task with multislice reading in CT. Med. Phys 44, 3990– 3999. 28555878 PMC5553707
  66. [72]Barrett HH, and Myers KJ (2004). Foundation of Image Science (John Wiley and Sons).
  67. [73]Burgess AE (1994). Statistically defined backgrounds: performance of a modified nonprewhitening observer model. J. Opt. Soc. Am. A Opt. Image Sci. Vis 11, 1237– 1242. 8189286
  68. [74]Barrett HH, Yao J, Rolland JP, and Myers KJ (1993). Model observers for assessment of image quality. Proc. Natl. Acad. Sci. USA 90, 9758– 9765. 8234311 PMC47653
  69. [76]Eckstein MP, and Whiting JS (1995). Lesion detection in structured noise. Acad. Radiol 2, 249– 253. 9419557
  70. [77]Simpson AL, Antonelli M, Bakas S, Bilello M, Farahani K, Van Ginneken B, Kopp-Schneider A, Landman BA, Litjens G, Menze B, (2019). A large annotated medical image dataset for the development and evaluation of segmentation algorithms. arXiv, arXiv:1902.09063v1. https://arxiv.org/abs/1902.09063. · arXiv

Cited by

Sorted by citations

Authors
Lago, M. A., Jonnalagadda, A., Abbey, C. K., Barufaldi, B. B., Bakic, P. R., Maidment, A. D. A., Leung, W. K., Weinstein, S. P., Englander, B. S., & Eckstein, M. P.
Affiliations
  • University of California, Santa Barbara (US)
  • University of Pennsylvania (US)
  • Sansum Medical Clinic (US)
Corresponding author
Eckstein, Miguel P.
ORCID
Venue
Current Biology
Publisher
Elsevier BV
Published
2021-03
Volume
31
Issue
5
Pages
1099-1106.e5
ISSN
0960-9822 (ISSN-L)
DOI
10.1016/j.cub.2020.12.029
PMC
PMC8048135
OpenAlex
W3124857197
Cited by
22
References
2
Citations by year
2026: 4 · 2025: 2 · 2024: 2 · 2023: 6 · 2022: 5 · 2021: 3
FWCI
2.09
Type
journal article
Language
en
Open access
Licence
Locations
  • Current Biology (publishedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • Army Research Office
  • National Institutes of Health
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
Keywords
breast cancer detection, Bayesian ideal observer, convolutional neural networks, eye movements, foveated vision, peripheral vision, radiologist performance
MeSH terms
Bayes Theorem, Eye Movements, Humans, Neural Networks, Computer, Phantoms, Imaging, Imaging, Three-Dimensional

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review