The DOI matches its Crossref record.
Unified measurement of observer performance in detecting and localizing target objects on images
Abstract
In this paper methods used to measure observer performance are reviewed, and a simple general model for finding and reporting target objects in gray‐scale image backgrounds is presented. That model provides the basis for a combined measurement of detection and localization performance in various image‐interpretation tasks, whether by human observers or by realized computer algorithms. The model assumes that (1) an observer's detection response and first choice of target location both depend on the “maximally suspicious” finding on an image, (2) a correct (first‐choice) localization of the actual target occurs if and only if its location is selected as the most suspicious, and (3) a target's presence does not alter the degree of suspicion engendered by any other (normal) image findings. Formalization of these assumptions relates the ROC curve, which measures the ability to discriminate between images containing targets and images without targets, to the “Localization Response” (LROC) curve, which measures the conjoint ability to detect and correctly localize the actual targets in those images. A maximum‐likelihood statistical procedure, developed for a two‐parameter “binormal” version of this model, concurrently fits both the ROC and LROC curves from an observer's image ratings and target localizations for a set of image interpretations. The model's application is illustrated (and compared to standard ROC analysis) using sets of rating and localization data from radiologists asked to search chest films for pulmonary nodules. This model is then extended to multiple‐report (“free‐response”) interpretations of multiple‐target images, under the stringent requirement that an observer's detection capability and criterion for reporting possible targets both remain stationary across images and across the successive reports made on a given image. That extended model yields formulations and predictions for the so‐called “Free‐Response” (FROC) curve, and for a recently proposed “Alternative FROC” (AFROC) curve. Tests of that model's “stationarity” assumptions are illustrated using radiologists' free‐search interpretations of chest films for pulmonary nodules, and they suggest that human observers may often violate those assumptions when making multiple‐report interpretations of images.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cited by
International evaluation of an AI system for breast cancer screening
McKinney, S. M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G. S., Darzi, A., et al. (2020) · Nature · doi:10.1038/s41586-019-1799-6 · Cited by 3,287
Kim, H.-E., Kim, H. H., Han, B.-K., Kim, K. H., Han, K., Nam, H., Lee, E. H., & Kim, E.-K. (2020) · The Lancet Digital Health · doi:10.1016/s2589-7500(20)30003-0 · Cited by 525
The psychophysics of visual search
Palmer, J., Verghese, P., & Pavel, M. (2000) · Vision Research · doi:10.1016/s0042-6989(99)00244-8 · Cited by 454
Zaidi, H., & El Naqa, I. (2010) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-010-1423-3 · Cited by 366
Observer studies involving detection and localization: Modeling, analysis, and validation
Chakraborty, D. P., & Berbaum, K. S. (2004) · Medical Physics · doi:10.1118/1.1769352 · Cited by 351
Current perspectives in medical image perception
Krupinski, E. A. (2010) · Attention, Perception & Psychophysics · doi:10.3758/app.72.5.1205 · Cited by 343
Obuchowski (2005) · doi:10.2214/ajr.184.2.01840364 · Cited by 321
Computer-Aided Detection and Diagnosis in Mammography
Sampat, M. P., Markey, M. K., & Bovik, A. C. (2005) · Handbook of Image and Video Processing · doi:10.1016/b978-012119792-6/50130-3 · Cited by 302
Impact of Time-of-Flight on PET Tumor Detection
Kadrmas, D. J., Casey, M. E., Conti, M., Jakoby, B. W., Lois, C., & Townsend, D. W. (2009) · Journal of Nuclear Medicine · doi:10.2967/jnumed.109.063016 · Cited by 277
Metz, C. E. (2006) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2006.02.021 · Cited by 274
Surti, S., Scheuermann, J., El Fakhri, G., Daube-Witherspoon, M. E., Lim, R., Abi-Hatem, N., Moussallem, E., Benard, F., Mankoff, D., & Karp, J. S. (2011) · Journal of Nuclear Medicine · doi:10.2967/jnumed.110.086678 · Cited by 104
Schaefferkoetter, J., Casey, M., Townsend, D., & El Fakhri, G. (2013) · Physics in Medicine and Biology · doi:10.1088/0031-9155/58/5/1465 · Cited by 85
Evaluating Imaging and Computer-aided Detection and Diagnosis Devices at the FDA
Gallas, B. D., Chan, H.-P., D’Orsi, C. J., Dodd, L. E., Giger, M. L., Gur, D., Krupinski, E. A., Metz, C. E., Myers, K. J., Obuchowski, N. A., et al. (2012) · Academic Radiology · doi:10.1016/j.acra.2011.12.016 · Cited by 81
Metadata
- Authors
- Swensson, R. G.
- Venue
- Medical Physics
- Publisher
- Wiley
- Published
- 1996-10
- Volume
- 23
- Issue
- 10
- Pages
- 1709-1725
- ISSN
- 0094-2405 (ISSN-L)
- 2473-4209
- OpenAlex
- W1971724710
- Cited by
- 286
- References
- 0
- FWCI
- 2.80
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.