Optical coherence tomography grading reproducibility during the Comparison of Age-related Macular Degeneration Treatments Trials
Open accessAbstract
Objective To report reading center reproducibility during grading of Stratus optical coherence tomography (OCT) (Carl Zeiss Meditec, Dublin, CA) images obtained during the Comparison of Age-Related Macular Degeneration Treatments Trials (CATT).Design Prospective, clinical trial.Participants Independent reading teams reevaluated 270 OCT scans randomly sampled from the first 2 years of CATT enrollment. To assess temporal drift, a cohort of 23 scans submitted during the initial portion of the CATT study was longitudinally followed with serial reproducibility analysis.Intervention The CATT readers performed standardized grading of OCT images. A reader team, composed of 2 independent readers and a senior reader, evaluated each scan. Grading included the CATT OCT end points of total thickness at the foveal center point and intraretinal fluid (IRF), subretinal fluid (SRF), and subretinal pigment epithelium (RPE) fluid. Independent reading teams masked to the results of initial grading reevaluated scans to determine the reproducibility of qualitative grading and measurements.Main outcome measures Categorical grading agreement was reported using percent agreement and kappa statistic, and measurement agreement was reported using intraclass correlations and paired differences.Results Reading center teams reproducibly graded IRF (percent agreement = 73%, kappa = 0.48; 95% confidence interval [CI], 0.38-0.58), SRF (percent agreement = 90%; kappa = 0.80; 95% CI, 0.73-0.87), and sub-RPE fluid (percent agreement 88%; kappa = 0.75; 95% CI, 0.67-0.83). For independent reading center team measurements of total thickness at the foveal center point, the intraclass correlation was 0.99 (95% CI, 0.99-0.99), and the mean paired difference between reading center teams was 4 μm (95% limits of agreement, -55 to 47 μm). There was no qualitative or quantitative grading drift.Conclusions The standardized protocols used to evaluate OCT scans from the CATT study were reproducible. The methods used are suitable to monitor OCT imaging data from a large, neovascular age-related macular degeneration, interventional, multicenter study.Financial disclosure(s) The author(s) have no proprietary or commercial interest in any materials discussed in this article.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The Director of Grading and the Reading Center Director stayed masked to subject identifiers and resolved reader disagreements.
“The Director of Grading (CAT) and the Reading Center Director (GJJ) remained masked to subject identifiers and made final decisions on reader disagreements that remained controversial after arbitration.17”
From Development and Course of Scars in the Comparison of Age-Related Macular Degeneration Treatments Trials (2018) · cites this paper as [17]
The Materials and Methods section (Team Based Grading; page unavailable) says that findings still controversial after arbitration were forwarded to the Director of Grading for a final decision. It does not say that the Reading Center Director also made those decisions or that either director remained masked to subject identifiers; its masking statement concerns the initial readers. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
High agreement but low Kappa: I. the problems of two paradoxes
Feinstein, A. R., & Cicchetti, D. V. (1990) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(90)90158-l · Cited by 3,034 · see citing works
Ranibizumab and bevacizumab for neovascular age-related macular degeneration
(2011) · New England Journal of Medicine · doi:10.1056/nejmoa1102673 · Cited by 2,669 · see citing works
* (1991) · Ophthalmology · doi:10.1016/s0161-6420(13)38012-9 · Cited by 2,058 · see citing works
Ranibizumab for Macular Edema following Central Retinal Vein Occlusion
Brown, D. M., Campochiaro, P. A., Singh, R. P., Li, Z., Gray, S., Saroj, N., Rundle, A. C., Rubio, R. G., & Murahashi, W. Y. (2010) · Ophthalmology · doi:10.1016/j.ophtha.2010.02.022 · Cited by 1,158 · see citing works
Intravitreal Bevacizumab (Avastin) for Neovascular Age-Related Macular Degeneration
Avery, R. L., Pieramici, D. J., Rabena, M., CASTELLARIN, A. A., Nasir, M. A., & Giust, M. J. (2006) · Ophthalmology · doi:10.1016/j.ophtha.2005.11.019 · Cited by 1,133 · see citing works
Haller, J. A., Bandello, F. M., Belfort, R., et al. (2010) · Ophthalmology · doi:10.1016/j.ophtha.2010.03.032 · Cited by 1,069 · see citing works
Fung, A. E., Lalwani, G. A., Rosenfeld, P. J., Dubovy, S. R., Michels, S., Feuer, W. J., Puliafito, C. A., Davis, J. L., Flynn, H. W., & Esquiabro, M. (2007) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2007.01.028 · Cited by 972 · see citing works
Ranibizumab for Macular Edema following Branch Retinal Vein Occlusion
Campochiaro, P. A., Heier, J. S., Feiner, L., Gray, S., Saroj, N., Rundle, A. C., Murahashi, W. Y., & Rubio, R. G. (2010) · Ophthalmology · doi:10.1016/j.ophtha.2010.02.021 · Cited by 821 · see citing works
Quantitative Assessment of Macular Edema With Optical Coherence Tomography
Hee, M. R. (1995) · Archives of Ophthalmology · doi:10.1001/archopht.1995.01100080071031 · Cited by 714 · see citing works
A General Methodology for the Analysis of Experiments with Repeated Measurement of Categorical Data
Koch, G. G., Landis, J. R., Freeman, J. L., Freeman, D. H., & Lehnen, R. G. (1977) · Biometrics · doi:10.2307/2529309 · Cited by 611 · see citing works
Optical coherence tomography to detect and manage retinal disease and glaucoma
Jaffe, G. J., & Caprioli, J. (2003) · American Journal of Ophthalmology · doi:10.1016/s0002-9394(03)00792-x · Cited by 508 · see citing works
Self-citationRich, R., Rosenfeld, P. J., Puliafito, C. A., Dubovy, S. R., Davis, J. L., Flynn, H. W., GONZALEZ, S., Feuer, W. J., Lin, R. C.-S., Lalwani, G. A., et al. (2006) · Retina · doi:10.1097/01.iae.0000225766.75009.3a · Cited by 484 · see citing works
Optical Coherence Tomography of Age-related Macular Degeneration and Choroidal Neovascularization
Hee, M. R., Baumal, C. R., Puliafito, C. A., Duker, J. S., Reichel, E., Wilkins, J. R., Coker, J. G., Schuman, J. S., Swanson, E. A., & Fujimoto, J. G. (1996) · Ophthalmology · doi:10.1016/s0161-6420(96)30512-5 · Cited by 413 · see citing works
Characterization of Epiretinal Membranes Using Optical Coherence Tomography
Wilkins, J. R., Puliafito, C. A., Hee, M. R., et al. (1996) · Ophthalmology · doi:10.1016/s0161-6420(96)30377-1 · Cited by 392 · see citing works
Srinivasan, V. J., Wojtkowski, M., Witkin, A. J., et al. (2006) · Ophthalmology · doi:10.1016/j.ophtha.2006.05.046 · Cited by 318 · see citing works
Ophthalmic imaging by spectral optical coherence tomography
Wojtkowski, M., Bajraszewski, T., Gorczynska, I. M., et al. (2004) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2004.04.049 · Cited by 292 · see citing works
A Treat and Extend Regimen Using Ranibizumab for Neovascular Age-Related Macular Degeneration
Gupta, O. P., Shienbaum, G., Patel, A. H., Fecarotta, C., Kaiser, R. S., & Regillo, C. D. (2010) · Ophthalmology · doi:10.1016/j.ophtha.2010.02.032 · Cited by 272 · see citing works
Kaiser, P. K., Blodi, B. A., Shapiro, H., & Acharya, N. R. (2007) · Ophthalmology · doi:10.1016/j.ophtha.2007.04.030 · Cited by 218 · see citing works
Spectral-Domain Optical Coherence Tomography for Monitoring Epiretinal Membrane Surgery
Falkner-Radler, C. I., Glittenberg, C. G., Hagen, S., et al. (2010) · Ophthalmology · doi:10.1016/j.ophtha.2009.08.034 · Cited by 196 · see citing works
Optical coherence tomography findings following photodynamic therapy of choroidal neovascularization
Rogers, A. H., Martidis, A., Greenberg, P. B., et al. (2002) · American Journal of Ophthalmology · doi:10.1016/s0002-9394(02)01566-0 · Cited by 184 · see citing works
Coscas, F., COSCAS, G. J., Souied, E. H., Tick, S., & Soubrane, G. (2007) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2007.06.014 · Cited by 164 · see citing works
Crewson, P. E. (2005) · American Journal of Roentgenology · doi:10.2214/ajr.184.5.01841391 · Cited by 156 · see citing works
Dadgostar, H., Ventura, A. A. C. M., Chung, J. Y., Sharma, S. K., & Kaiser, P. K. (2009) · Ophthalmology · doi:10.1016/j.ophtha.2009.05.033 · Cited by 124 · see citing works
Ting, T. D. (2002) · Archives of Ophthalmology · doi:10.1001/archopht.120.6.731 · Cited by 109 · see citing works
Rothenbuehler, S. P., Waeber, D., Brinkmann, C. K., Wolf, S., & Wolf, S. (2009) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2008.12.005 · Cited by 99 · see citing works
Sayanagi, K., Sharma, S. K., Yamamoto, T., & Kaiser, P. K. (2009) · Ophthalmology · doi:10.1016/j.ophtha.2008.11.002 · Cited by 90 · see citing works
Krebs, I., Hagen, S., Brannath, W., Haas, P., Womastek, I., Salvo, G. D., Ansari‐Shahrezaei, S., & Binder, S. (2010) · Ophthalmology · doi:10.1016/j.ophtha.2010.04.032 · Cited by 45 · see citing works
Keane, P. A., Bhatti, R. A., Brubaker, J. W., et al. (2009) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2009.03.004 · Cited by 42 · see citing works
Optical Coherence Tomography Reader Agreement in Neovascular Age-related Macular Degeneration
Zhang, N., Hoffmeyer, G. C., Young, E., et al. (2007) · American Journal of Ophthalmology · doi:10.1016/j.ajo.2007.03.056 · Cited by 36 · see citing works
Ritter, M., Elledge, J. A., Simader, C., Deák, G. G., Benesch, T., Blodi, B. A., & Schmidt-Erfurth, U. M. (2010) · British Journal of Ophthalmology · doi:10.1136/bjo.2009.175976 · Cited by 36 · see citing works
Glassman, A. R., Beck, R. W., Browning, D. J., Danis, R. P., & Kollman, C. (2009) · Investigative Ophthalmology & Visual Science · doi:10.1167/iovs.08-1881 · Cited by 33 · see citing works
Domalpally, A. (2009) · Archives of Ophthalmology · doi:10.1001/archophthalmol.2009.277 · Cited by 26 · see citing works
Topographic Angiography and Optical Coherence Tomography: A Correlation of Imaging Characteristics
Ahlers, C., Michels, S., Elsner, H., et al. (2005) · European Journal of Ophthalmology · doi:10.1177/112067210501500619 · Cited by 10 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Maguire, M. G., Martin, D. F., Ying, G.-S., Jaffe, G. J., Daniel, E., Grunwald, J. E., Toth, C. A., Ferris, F. L., & Fine, S. L. (2016) · Ophthalmology · doi:10.1016/j.ophtha.2016.03.045 · Cited by 708 · see citing works
Self-citationRisk of geographic atrophy in the comparison of age-related macular degeneration treatments trials
Grunwald, J. E., Daniel, E., Huang, J., Ying, G.-S., Maguire, M. G., Toth, C. A., Jaffe, G. J., Fine, S. L., Blodi, B., Klein, M. L., et al. (2014) · Ophthalmology · doi:10.1016/j.ophtha.2013.08.015 · Cited by 590 · see citing works
Self-citationRisk of Scar in the Comparison of Age-related Macular Degeneration Treatments Trials
Daniel, E., Toth, C. A., Grunwald, J. E., Jaffe, G. J., Martin, D. F., Fine, S. L., Huang, J., Ying, G.-S., Hagstrom, S. A., Winter, K., et al. (2014) · Ophthalmology · doi:10.1016/j.ophtha.2013.10.019 · Cited by 335 · see citing works
Self-citationJaffe, G. J., Martin, D. F., Toth, C. A., Daniel, E., Maguire, M. G., Ying, G.-S., Grunwald, J. E., & Huang, J. (2013) · Ophthalmology · doi:10.1016/j.ophtha.2013.01.073 · Cited by 294 · see citing works
Self-citationSharma, S., Toth, C. A., Daniel, E., Grunwald, J. E., Maguire, M. G., Ying, G.-S., Huang, J., Martin, D. F., & Jaffe, G. J. (2016) · Ophthalmology · doi:10.1016/j.ophtha.2015.12.002 · Cited by 231 · see citing works
Self-citationGrunwald, J. E., Pistilli, M., Daniel, E., Ying, G.-S., Pan, W., Jaffe, G. J., Toth, C. A., Hagstrom, S. A., Maguire, M. G., & Martin, D. F. (2017) · Ophthalmology · doi:10.1016/j.ophtha.2016.09.012 · Cited by 214 · see citing works
Self-citationDaniel, E., Pan, W., Ying, G.-S., Kim, B. J., Grunwald, J. E., Ferris, F. L., Jaffe, G. J., Toth, C. A., Martin, D. F., Fine, S. L., et al. (2018) · Ophthalmology · doi:10.1016/j.ophtha.2018.01.004 · Cited by 119 · see citing works
Self-citationMaguire, M. G., Daniel, E., Shah, A. R., Grunwald, J. E., Hagstrom, S. A., Avery, R. L., Huang, J., Martin, R. W., Roth, D. B., Castellarin, A. A., et al. (2013) · Ophthalmology · doi:10.1016/j.ophtha.2013.03.017 · Cited by 93 · see citing works
Retinal specialist versus artificial intelligence detection of retinal fluid from OCT
Keenan, T. D. L., Clemons, T. E., Domalpally, A., Elman, M. J., Havilio, M., Agrón, E., Benyamini, G., & Chew, E. Y. (2021) · Ophthalmology · doi:10.1016/j.ophtha.2020.06.038 · Cited by 93 · see citing works
Altaweel, M. M., Daniel, E., Martin, D. F., Mittra, R. A., Grunwald, J. E., Lai, M. M., Melamud, A., Morse, L. S., Huang, J., Ferris, F. L., et al. (2015) · Ophthalmology · doi:10.1016/j.ophtha.2014.08.020 · Cited by 51 · see citing works
Self-citation
- Authors
- DeCroos, F. C., Toth, C. A., Stinnett, S. S., Heydary, C. S., Burns, R., & Jaffe, G. J.
- Affiliations
- Duke University (US)
- Mid Atlantic Retina (US)
- Corresponding author
- Jaffe, Glenn J.
- Venue
- Ophthalmology
- Publisher
- Elsevier BV
- Published
- 2012-12
- Volume
- 119
- Issue
- 12
- Pages
- 2549-2557
- ISSN
- 0161-6420 (ISSN-L)
- PMC
- PMC3631582
- OpenAlex
- W2029010710
- Cited by
- 67
- References
- 35
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 2 · 2023: 5 · 2022: 6 · 2021: 2 · 2020: 4 · 2019: 4 · 2018: 5 · 2017: 2 · 2016: 6 · 2015: 6 · 2014: 12 · 2013: 7
- FWCI
- 7.47
- Type
- journal article
- Language
- en
- Open access
- green
- europepmc.org
- Locations
- Ophthalmology (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Funders
- Heed Ophthalmic Foundation
- Topics
- Health Sciences › Medicine › Ophthalmology › Retinal Diseases and Treatments
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- optical coherence tomography, age-related macular degeneration, neovascular AMD, interobserver agreement, intraretinal fluid, subretinal fluid, temporal drift, Stratus OCT
- MeSH terms
- Bevacizumab, Ranibizumab, Double-Blind Method, Humans, Macular Degeneration, Prospective Studies, Quality Control, Retina, Visual Acuity, Reproducibility of Results, Observer Variation, Epiretinal Membrane, Angiogenesis Inhibitors, Tomography, Optical Coherence, Vascular Endothelial Growth Factor A, Subretinal Fluid, Antibodies, Monoclonal, Humanized
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
