Inter-rater agreement in glioma segmentations on longitudinal MRI
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Tumor segmentation of glioma on MRI is a technique to monitor, quantify and report disease progression. Manual MRI segmentation is the gold standard but very labor intensive. At present the quality of this gold standard is not known for different stages of the disease, and prior work has mainly focused on treatment-naive glioblastoma. In this paper we studied the inter-rater agreement of manual MRI segmentation of glioblastoma and WHO grade II-III glioma for novices and experts at three stages of disease. We also studied the impact of inter-observer variation on extent of resection and growth rate. METHODS: In 20 patients with WHO grade IV glioblastoma and 20 patients with WHO grade II-III glioma (defined as non-glioblastoma) both the enhancing and non-enhancing tumor elements were segmented on MRI, using specialized software, by four novices and four experts before surgery, after surgery and at time of tumor progression. We used the generalized conformity index (GCI) and the intra-class correlation coefficient (ICC) of tumor volume as main outcome measures for inter-rater agreement. RESULTS: For glioblastoma, segmentations by experts and novices were comparable. The inter-rater agreement of enhancing tumor elements was excellent before surgery (GCI 0.79, ICC 0.99) poor after surgery (GCI 0.32, ICC 0.92), and good at progression (GCI 0.65, ICC 0.91). For non-glioblastoma, the inter-rater agreement was generally higher between experts than between novices. The inter-rater agreement was excellent between experts before surgery (GCI 0.77, ICC 0.92), was reasonable after surgery (GCI 0.48, ICC 0.84), and good at progression (GCI 0.60, ICC 0.80). The inter-rater agreement was good between novices before surgery (GCI 0.66, ICC 0.73), was poor after surgery (GCI 0.33, ICC 0.55), and poor at progression (GCI 0.36, ICC 0.73). Further analysis showed that the lower inter-rater agreement of segmentation on postoperative MRI could only partly be explained by the smaller volumes and fragmentation of residual tumor. The median interquartile range of extent of resection between raters was 8.3% and of growth rate was 0.22 mm/year. CONCLUSION: Manual tumor segmentations on MRI have reasonable agreement for use in spatial and volumetric analysis. Agreement in spatial overlap is of concern with segmentation after surgery for glioblastoma and with segmentation of non-glioblastoma by non-experts.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Cicchetti, D. V. (1994) · Psychological Assessment · doi:10.1037/1040-3590.6.4.284 · Cited by 9,349
Forming inferences about some intraclass correlation coefficients.
McGraw, K. O., & Wong, S. P. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.1.30 · Cited by 6,903
The multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781
The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository
Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., et al. (2013) · Journal of Digital Imaging · doi:10.1007/s10278-013-9622-7 · Cited by 4,599 · cited 1×
“To facilitate MRI interpretation, raters were acquainted with the VASARI-criteria (Visually AcceSAble Rembrandt Images, as proposed by The Cancer Imaging Archive (Clark et al., 2013)), but no segmentations rules were imposed.”
Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J. S., Freymann, J. B., Farahani, K., & Davatzikos, C. (2017) · Scientific Data · doi:10.1038/sdata.2017.117 · Cited by 3,049
Pallud, J., Taillandier, L., Capelle, L., Fontaine, D., Peyre, M., Ducray, F., Duffau, H., & Mandonnet, E. (2012) · Neurosurgery · doi:10.1227/neu.0b013e31826213de · Cited by 135 · cited 2×
“…2008), quantification of the effects of (pharmacological) interventions (Ben Abdallah et al., 2018b; Mandonnet et al., 2010; Pallud et al., 2012a; Pallud et al., 2012b), and statistical maps of care (De Witt Hamer et al., 2013; Mandonnet et al., 2007) and disease mechanisms (Amelot et al., 2017;…”
Background
Cited by
Kvasir-SEG: A Segmented Polyp Dataset
Jha, D., Smedsrud, P. H., Riegler, M. A., Halvorsen, P., de Lange, T., Johansen, D., & Johansen, H. D. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-37734-2_37 · Cited by 1,661
Karschnia, P., Young, J. S., Dono, A., Häni, L., Sciortino, T., Bruno, F., Juenger, S. T., Teske, N., Morshed, R. A., Haddad, A. F., et al. (2023) · Neuro-Oncology · doi:10.1093/neuonc/noac193 · Cited by 384
Srinivas, C., K. S., N. P., Zakariah, M., Alothaibi, Y. A., Shaukat, K., Partibane, B., & Awal, H. (2022) · Journal of Healthcare Engineering · doi:10.1155/2022/3264367 · Cited by 344
Data Augmentation for Brain-Tumor Segmentation: A Review
Nalepa, J., Marcinkiewicz, M., & Kawulok, M. (2019) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2019.00083 · Cited by 340
Lehtola, S. J., Tuulari, J. J., Scheinin, N. M., Karlsson, L., Parkkola, R., Merisaari, H., Lewis, J. D., Fonov, V. S., Louis Collins, D., Evans, A., et al. (2020) · NeuroImage: Clinical · doi:10.1016/j.nicl.2020.102380 · Cited by 51
Metadata
- Authors
- Visser, M., Müller, D. M. J., van Duijn, R. J. M., Smits, M., Verburg, N., Hendriks, E. J., Nabuurs, R. J. A., Bot, J. C. J., Eijgelaar, R. S., Witte, M., van Herk, M. B., Barkhof, F., de Witt Hamer, P. C., & de Munck, J. C.
- Venue
- NeuroImage: Clinical
- Publisher
- Elsevier BV
- Published
- 2019
- Volume
- 22
- Pages
- 101727
- ISSN
- 2213-1582 (ISSN-L)
- OpenAlex
- W2916758913
- Cited by
- 126
- References
- 7
- FWCI
- 5.46
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Inter-rater agreement in glioma segmentations on longitudinal MRI. https://app.cochat.ai/papers/10.1016/j.nicl.2019.101727
- The paper
- Visser, M., Müller, D. M. J., van Duijn, R. J. M., Smits, M., Verburg, N., Hendriks, E. J., Nabuurs, R. J. A., Bot, J. C. J., Eijgelaar, R. S., Witte, M., van Herk, M. B., Barkhof, F., de Witt Hamer, P. C., & de Munck, J. C. (2019). Inter-rater agreement in glioma segmentations on longitudinal MRI. NeuroImage: Clinical. https://doi.org/10.1016/j.nicl.2019.101727
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.