Formative Evaluation of Standardized Training for CT Colonographic Image Interpretation by Novice Readers
Abstract
PURPOSE: To introduce an educational intervention-specifically, a specialized training course-and perform a formative evaluation of the effect of the intervention on novice reader interpretation of computed tomography (CT) colonographic data. MATERIALS AND METHODS: The study was institutional review board approved. Ten normal and 50 abnormal cases, those of 60 patients with 93 polyps-61 polyps 6-9 mm in diameter and 32 polyps 10 mm or larger-were selected from a previously published trial. Seven novice readers underwent initial training that consisted of a 1-day course, reading assignments, a self-study computer module (with 61 limited data sets), observation of an expert interpreting three cases, and full interpretation of 10 cases with unblinding after each case. After training, the observers independently interpreted 60 cases by means of primary two-dimensional reading with unblinding after each case. For each case, the reading time and the location and maximal diameter of the polyp(s) were recorded. A t test was used to evaluate the observers' improvements, and empirical receiver operating characteristic (ROC) curves were constructed. RESULTS: By-patient sensitivities and specificities were determined for each observer. The lowest by-patient sensitivity at the 6 mm or larger polyp threshold was 86%, with 90% specificity. Four observers had 100% by-patient sensitivity at the 10 mm or larger polyp threshold, with 82%-97% specificity. For polyps 10 mm or larger, mean sensitivity and specificity were 98% and 92%, respectively. For the last 20 cases, the average interpretation time per case was 25 minutes. The range of areas under the ROC curve across observers was low: 0.86-0.95. CONCLUSION: In the described polyp-enriched cohort, novice CT colonographic data readers achieved high sensitivity and good specificity at formative evaluation of a comprehensive training program. Use of a similar comprehensive training method might reduce interreader variability in interpretation accuracy and be useful for reader certification.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reports indicate enhanced X-ray diagnostic techniques by technicians in medically underserved areas such as South Africa and Australia, as well as specialised diagnostic improvements for CT colonographic data, endometrial tumours and malignant skin tumours.
“Reports indicate enhanced X-ray diagnostic techniques by technicians in medically underserved areas such as South Africa18 and Australia,19 as well as specialised diagnostic improvements for CT colonographic data,20 endometrial tumours21 and malignant skin tumours.22”
From Using artificial intelligence to improve human performance: efficient retinal disease detection training with synthetic images (2024) · cites this paper as [20]
Cites
Computed Tomographic Virtual Colonoscopy to Screen for Colorectal Neoplasia in Asymptomatic Adults
Pickhardt, P. J., Choi, J.-Y., Hwang, I., Butler, J. A., Puckett, M. L., Hildebrandt, H. A., Wong, R. K. H., Nugent, P. A., Mysliwiec, P. A., & Schindler, W. R. (2003) · New England Journal of Medicine · doi:10.1056/nejmoa031618 · Cited by 1,963 · see citing works
Computed Tomographic Colonography (Virtual Colonoscopy)
Cotton, P. B., Durkalski, V., Pineau, B. C., et al. (2004) · JAMA · doi:10.1001/jama.291.14.1713 · Cited by 676 · see citing works
CT Colonography Reporting and Data System: A Consensus Proposal
Zalis, M. E., Barish, M. A., Choi, J. R., et al. (2005) · Radiology · doi:10.1148/radiol.2361041926 · Cited by 610 · see citing works
Rockey, D., Paulson, E. K., Niedzwiecki, D., et al. (2005) · The Lancet · doi:10.1016/s0140-6736(05)17784-8 · Cited by 574 · see citing works
Colorectal Neoplasia: Performance Characteristics of CT Colonography for Detection in 300 Patients
Yee, J., Akerkar, G. A., Hung, R. K., Steinauer-Gebauer, A. M., Wall, S. D., & McQuaid, K. R. (2001) · Radiology · doi:10.1148/radiology.219.3.r01jn40685 · Cited by 486 · see citing works
Location of Adenomas Missed by Optical Colonoscopy
Pickhardt, P. J., Nugent, P. A., Mysliwiec, P. A., Choi, J.-Y., & Schindler, W. R. (2004) · Annals of Internal Medicine · doi:10.7326/0003-4819-141-5-200409070-00009 · Cited by 445 · see citing works
Johnson, C. D., Harmsen, W. S., Wilson, L. A., MacCarty, R. L., Welch, T. J., Ilstrup, D. M., & Ahlquist, D. A. (2003) · Gastroenterology · doi:10.1016/s0016-5085(03)00894-1 · Cited by 349 · see citing works
Optimization of CT Colonography Technique: Prospective Trial in 180 Patients
Fletcher, J. G., Johnson, C. D., Welch, T. J., et al. (2000) · Radiology · doi:10.1148/radiology.216.3.r00au41704 · Cited by 344 · see citing works
Self-citationBeyond Formative and Summative Evaluation
Scriven, M. (1991) · Teachers College Record The Voice of Scholarship in Education · Cited by 187 · see citing works
Screening CT Colonography: How I Do It
Pickhardt, P. J. (2007) · American Journal of Roentgenology · doi:10.2214/ajr.07.2136 · Cited by 173 · see citing works
Pickhardt, P. J., TAYLOR, A. J., Kim, D. H., et al. (2006) · Radiology · doi:10.1148/radiol.2412052007 · Cited by 163 · see citing works
Detection of flat lesions in the colon with CT colonography
Fidler, J. L., Johnson, C. D., MacCarty, R. L., Welch, T. L., Hara, A. K., & Harmsen, W. S. (2002) · Abdominal Imaging · doi:10.1007/s00261-001-0171-z · Cited by 150 · see citing works
Spinzi, G., Belloni, G., Martegani, A., et al. (2001) · The American Journal of Gastroenterology · doi:10.1111/j.1572-0241.2001.03550.x · Cited by 145 · see citing works
Assessment of Medical Imaging Systems and Computer Aids: A Tutorial Review
Wagner, R. F., Metz, C. E., & Campbell, G. (2007) · Academic Radiology · doi:10.1016/j.acra.2007.03.001 · Cited by 142 · see citing works
Pickhardt, P. J., Nugent, P. A., Choi, J. R., et al. (2004) · American Journal of Roentgenology · doi:10.2214/ajr.183.5.1831343 · Cited by 134 · see citing works
CT colonography: effect of experience and training on reader performance
Taylor, S. A., Halligan, S., Burling, D. N., et al. (2004) · European Radiology · doi:10.1007/s00330-004-2262-z · Cited by 119 · see citing works
Consensus on Current Clinical Practice of Virtual Colonoscopy
Barish, M. A., Soto, J. A., & Ferrucci, J. T. (2005) · American Journal of Roentgenology · doi:10.2214/ajr.184.3.01840786 · Cited by 113 · see citing works
Detection of colorectal lesions with virtual computed tomographic colonography
Laghi, A., Iannaccone, R., Carbone, I., et al. (2002) · The American Journal of Surgery · doi:10.1016/s0002-9610(01)00857-1 · Cited by 112 · see citing works
Pickhardt, P. J., Lee, A., McFarland, E. G., et al. (2005) · Radiology · doi:10.1148/radiol.2363041534 · Cited by 102 · see citing works
Baker, M. E., Bogoni, L., Obuchowski, N. A., Dass, C., Kendzierski, R. M., Remer, E. M., Einstein, D. M., Cathier, P., Jerebko, A. K., Lakare, S., et al. (2007) · Radiology · doi:10.1148/radiol.2451061116 · Cited by 87 · see citing works
Rockey, D. C., Barish, M. A., Brill, J. V., et al. (2007) · Gastroenterology · doi:10.1053/j.gastro.2007.06.001 · Cited by 83 · see citing works
CT Colonography: False-Negative Interpretations
Doshi, T., Rusinak, D. J., Halvorsen, R. A., et al. (2007) · Radiology · doi:10.1148/radiol.2441061122 · Cited by 83 · see citing works
Effect of Directed Training on Reader Performance for CT Colonography: Multicenter Study
(2007) · Radiology · doi:10.1148/radiol.2421051000 · Cited by 69 · see citing works
Stool Tagging Applied in Thin-slice Multidetector Computed Tomography Colonography
Thomeer, M. G. J., Carbone, I., Bosmans, H., et al. (2003) · Journal of Computer Assisted Tomography · doi:10.1097/00004728-200303000-00005 · Cited by 61 · see citing works
Nonradiologists as second readers for intraluminal findings at CT colonography1
Bodily, K. D., Fletcher, J. G., Engelby, T. A., et al. (2004) · Academic Radiology · doi:10.1016/j.acra.2004.10.055 · Cited by 61 · see citing works
Self-citationSignificance of Missed Polyps at CT Colonography
Macari, M., Bini, E. J., Jacobs, S. L., et al. (2004) · American Journal of Roentgenology · doi:10.2214/ajr.183.1.1830127 · Cited by 57 · see citing works
Reader error during CT colonography: causes and implications for training
Slater, A., Taylor, S. A., Tam, E., et al. (2006) · European Radiology · doi:10.1007/s00330-006-0299-x · Cited by 56 · see citing works
(2003) · Springer eBooks · doi:10.1007/978-0-387-21558-7 · Cited by 53 · see citing works
Understanding interpretive errors in radiologists learning computed tomography colonography1
Fidler, J. L., Fletcher, J. G., Johnson, C. D., et al. (2004) · Academic Radiology · doi:10.1016/j.acra.2004.03.052 · Cited by 53 · see citing works
Self-citationAdvice for Optimizing Colonic Distention and Minimizing Risk of Perforation during CT Colonography
Dachman, A. H. (2006) · Radiology · doi:10.1148/radiol.2392051374 · Cited by 53 · see citing works
Self-citationEffect of investigator experience in CT colonography
Gluecker, T. M., Meuwly, J., Pescatore, P., et al. (2002) · European Radiology · doi:10.1007/s00330-001-1280-3 · Cited by 50 · see citing works
CT Colonography: Visualization Methods, Interpretation, and Pitfalls
Dachman, A. H., Lefere, P. A., Gryspeerdt, S. S., et al. (2007) · Radiologic Clinics of North America · doi:10.1016/j.rcl.2007.03.007 · Cited by 44 · see citing works
Self-citationReader Training in CT Colonography: How Much Is Enough?
Soto, J. A., Barish, M. A., & Yee, J. (2005) · Radiology · doi:10.1148/radiol.2371050003 · Cited by 43 · see citing works
Performance of Radiographers in the Evaluation of CT Colonographic Images
Jensch, S., van Gelder, R. E., Florie, J., et al. (2007) · American Journal of Roentgenology · doi:10.2214/ajr.06.0451 · Cited by 41 · see citing works
CT colonography: interpretative performance in a non-academic environment
Burling, D. N., Halligan, S., Atchley, J., et al. (2007) · Clinical Radiology · doi:10.1016/j.crad.2006.11.004 · Cited by 36 · see citing works
Pickhardt, P. J., & Choi, J. R. (2005) · American Journal of Roentgenology · doi:10.2214/ajr.184.5.01841581 · Cited by 22 · see citing works
Perceptive errors in CT colonography
Nio, C. Y., de Vries, A. H., & Stoker, J. (2006) · Abdominal Imaging · doi:10.1007/s00261-006-9170-4 · Cited by 16 · see citing works
Halligan, S., Taylor, S. A., & Burling, D. N. (2004) · JAMA · doi:10.1001/jama.292.4.432-a · Cited by 13 · see citing works
Causes of False-Negative Findings at CT Colonography
Halligan, S. (2006) · Radiology · doi:10.1148/radiol.2383051049 · Cited by 11 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Tabuchi, H., Engelmann, J., Maeda, F., Nishikawa, R., Nagasawa, T., Yamauchi, T., Tanabe, M., Akada, M., Kihara, K., Nakae, Y., et al. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-324923 · Cited by 22 · see citing works
- Authors
- Dachman, A. H., Kelly, K. B., Zintsmaster, M. P., Rana, R., Khankari, S., Novak, J. D., Ali, A. N., Qalbani, A., & Fletcher, J. G.
- Affiliations
- University of Chicago (US)
- Mayo Clinic (US)
- Venue
- Radiology
- Published
- 2008-10
- Volume
- 249
- Issue
- 1
- Pages
- 167-177
- ISSN
- 0033-8419 (ISSN-L)
- 1527-1315
- OpenAlex
- W2044571035
- Cited by
- 44
- References
- 39
- Citations by year
- 2025: 1 · 2024: 1 · 2023: 1 · 2022: 1 · 2018: 1 · 2017: 1 · 2015: 1 · 2014: 2 · 2013: 1 · 2012: 8
- FWCI
- 4.22
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Radiology (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Health Sciences › Medicine › Oncology › Colorectal Cancer Screening and Detection
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiology practices and education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Innovations in Medical Education
- Keywords
- CT colonography, polyp detection, radiology education, beginning readers, formative assessment, interobserver variability, colorectal cancer screening
- MeSH terms
- Colonic Polyps, Education, Medical, Continuing, Humans, Sensitivity and Specificity, Teaching, Colonography, Computed Tomographic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
