Comparative evaluation of autocontouring in clinical practice: A practical method using the Turing test
Open accessAbstract
Purpose: Automated techniques for estimating the contours of organs and structures in medical images have become more widespread and a variety of measures are available for assessing their quality. Quantitative measures of geometric agreement, for example, overlap with a gold‐standard delineation, are popular but may not predict the level of clinical acceptance for the contouring method. Therefore, surrogate measures that relate more directly to the clinical judgment of contours, and to the way they are used in routine workflows, need to be developed. The purpose of this study is to propose a method (inspired by the Turing Test) for providing contour quality measures that directly draw upon practitioners’ assessments of manual and automatic contours. This approach assumes that an inability to distinguish automatically produced contours from those of clinical experts would indicate that the contours are of sufficient quality for clinical use. In turn, it is anticipated that such contours would receive less manual editing prior to being accepted for clinical use. In this study, an initial assessment of this approach is performed with radiation oncologists and therapists. Methods: Eight clinical observers were presented with thoracic organ‐at‐risk contours through a web interface and were asked to determine if they were automatically generated or manually delineated. The accuracy of the visual determination was assessed, and the proportion of contours for which the source was misclassified recorded. Contours of six different organs in a clinical workflow were for 20 patient cases. The time required to edit autocontours to a clinically acceptable standard was also measured, as a gold standard of clinical utility. Established quantitative measures of autocontouring performance, such as Dice similarity coefficient with respect to the original clinical contour and the misclassification rate accessed with the proposed framework, were evaluated as surrogates of the editing time measured. Results: The mis classification rates for each organ were: esophagus 30.0%, heart 22.9%, left lung 51.2%, right lung 58.5%, mediastinum envelope 43.9%, and spinal cord 46.8%. The time savings resulting from editing the autocontours compared to the standard clinical workflow were 12%, 25%, 43%, 77%, 46%, and 50%, respectively, for these organs. The median Dice similarity coefficients between the clinical contours and the autocontours were 0.46, 0.90, 0.98, 0.98, 0.94, and 0.86, respectively, for these organs. Conclusions: A better correspondence with time saving was observed for the misclassification rate than the quantitative contour measures explored. From this, we conclude that the inability to accurately judge the source of a contour indicates a reduced need for editing and therefore a greater time saving overall. Hence, task‐based assessments of contouring performance may be considered as an additional way of evaluating the clinical utility of autosegmentation methods.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A blinded side-by-side comparison, akin to a comparative Turing test, was performed to assess an experienced operator's ability to discriminate between manual and automated outputs for the 60 multi-class test cases.
“Finally, a blinded comparison was performed to assess an experienced operator’s (GS) ability to discriminate between the manual and automated outputs using a side-by-side comparison, akin to a comparative Turing test [39] for the 60 multi-class test cases.”
From Fully Automated 3-D Ultrasound Segmentation of the Placenta, Amniotic Fluid, and Fetus for Early Pregnancy Assessment (2021) · cites this paper as [39]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · see citing works
Irshad, H., Veillard, A., Roux, L., & Racoceanu, D. (2014) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2013.2295804 · Cited by 641 · see citing works
Torsten Rohlfing, B., Brandt, R., Menzel, R., & Maurer, C. R. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2003.11.010 · Cited by 564 · see citing works
Pinar Saygin, A., Cicekli, I., & Akman, V. (2000) · Minds and Machines · doi:10.1023/a:1011288000451 · Cited by 383 · see citing works
Vision 20/20: Perspectives on automated image segmentation for radiotherapy
Sharp, G. C., Fritscher, K. D., Pekar, V., Peroni, M., Shusharina, N., Veeraraghavan, H., & Yang, J. (2014) · Medical Physics · doi:10.1118/1.4871620 · Cited by 377 · see citing works
Clinical evaluation of atlas and deep learning based automatic contouring for lung cancer
Lustberg, T., van Soest, J., Gooding, M. J., Peressutti, D., Aljabar, P., van der Stoep, J., van Elmpt, W., & Dekker, A. L. A. J. (2017) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2017.11.012 · Cited by 368 · see citing works
Self-citationLessons from a Restricted Turing Test
Shieber, S. M. (1994) · arXiv · doi:10.48550/arxiv.cmp-lg/9404002 · Cited by 222 · see citing works
Teguh, D. N., Levendag, P. C., Voet, P. W. J., Al‐Mamgani, A., Han, X., Wolf, T. K., Hibbard, L. S., Nowak, P. J. C. M., Akhiat, H., Dirkx, M. L. P., et al. (2010) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2010.07.009 · Cited by 193 · see citing works
Automatic Segmentation of Whole Breast Using Atlas Approach and Deformable Image Registration
Reed, V. K., Woodward, W. A., Zhang, L., Strom, E. A., Perkins, G. H., Tereffe, W., Oh, J., Yu, T. K., Bedrosian, I., Whitman, G. J., et al. (2008) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2008.07.001 · Cited by 121 · see citing works
Voet, P. W. J., Dirkx, M. L. P., Teguh, D. N., Hoogeman, M. S., Levendag, P. C., & Heijmen, B. J. M. (2011) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2010.11.017 · Cited by 114 · see citing works
An Automatic Mass Detection System in Mammograms Based on Complex Texture Features
Tai, S., Chen, Z.-S., & Tsai, W.-T. (2013) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2013.2279097 · Cited by 100 · see citing works
Delpon, G., Escande, A., Ruef, T., Darréon, J., Fontaine, J., Noblet, C., Supiot, S., Lacornerie, T., & Pasquier, D. (2016) · Frontiers in Oncology · doi:10.3389/fonc.2016.00178 · Cited by 86 · see citing works
GUNDERSON, K. (1964) · Mind · doi:10.1093/mind/lxxiii.290.234 · Cited by 72 · see citing works
Recommendations on how to establish evidence from auto-segmentation software in radiotherapy
Valentini, V., Boldrini, L., Damiani, A., & Muren, L. P. (2014) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2014.09.014 · Cited by 64 · see citing works
Sjöberg, C., Lundmark, M., Granberg, C., Johansson, S., Ahnesjö, A., & Montelius, A. (2013) · Radiation Oncology · doi:10.1186/1748-717x-8-229 · Cited by 54 · see citing works
Evaluation of atlas‐based auto‐segmentation software in prostate cancer patients
Greenham, S., Dean, J., Fu, C. K. K., Goman, J., Mulligan, J., Tune, D., Sampson, D., Westhuyzen, J., & McKay, M. J. (2014) · Journal of Medical Radiation Sciences · doi:10.1002/jmrs.64 · Cited by 51 · see citing works
Multiatlas segmentation of thoracic and abdominal anatomy with level set‐based local search
Schreibmann, E., Marcus, D. M., & Fox, T. H. (2014) · Journal of Applied Clinical Medical Physics · doi:10.1120/jacmp.v15i4.4468 · Cited by 46 · see citing works
Automated CT-based segmentation and quantification of total intracranial volume
Aguilar, C., Edholm, K., Simmons, A., Cavallin, L., Müller, S., Skoog, I. G., Larsson, E., Axelsson, R., Wahlund, L., & Westman, E. (2015) · European Radiology · doi:10.1007/s00330-015-3747-7 · Cited by 24 · see citing works
The impact of the number of atlases on the performance of automatic multi-atlas contouring
Larrue, A., Gujral, D. M., Nutting, C. M., & Gooding, M. J. (2015) · Physica Medica · doi:10.1016/j.ejmp.2015.10.020 · Cited by 17 · see citing works
Self-citationMulticenter Clinical Assessment of DIR Atlas-Based Autocontouring
Gooding, M. J., Chu, K.-Y., Conibear, J., Dilling, T. J., Durrant, L., Fuß, M., Gujral, D. M., Ove, R., Stevens, C. W., & Kadir, T. (2013) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2013.06.1892 · Cited by 9 · see citing works
Self-citationGautam, A., Weiss, E., Williamson, J. F., Ford, J. C., Sleeman, W. C., Jan, N., Saraiya, S., Orton, M. D., Zhang, L., & Murphy, M. J. (2013) · Medical Physics · doi:10.1118/1.4813955 · Cited by 5 · see citing works
Mattiucci, G. C., Placidi, L., Boldrini, L., Azario, L., Piccari, D., Mazzarella, C., Pasini, D., Dinapoli, N., Chiesa, S., & Valentini, V. (2014) · Radiotherapy and Oncology · doi:10.1016/s0167-8140(15)30238-3 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Gunderson K. The imitation game. Mind. 1964;73:234-245.
- Harnad S. The Turing Test is not a trick: Turing indistinguishability is a scientific criterion. 1992;3:9-10.
- Aguilar C, Edholm K, Simmons A, et al. To develop an algorithm to segment and obtain an estimate of total intracranial volume (tICV) from computed tomography (CT) images. Eur Radiol. 2015;25:3151-3160.
Cited by
Sorted by citations
Metrics reloaded: recommendations for image analysis validation
Maier‐Hein, L., Reinke, A., Godau, P., Tizabi, M. D., Buettner, F., Christodoulou, E., Glocker, B., Isensee, F., Kleesiek, J., Kozubek, M., et al. (2024) · Nature Methods · doi:10.1038/s41592-023-02151-z · Cited by 539 · see citing works
Vandewinckele, L., Claessens, M., Dinkla, A. M., Brouwer, C. L., Crijns, W., Verellen, D., & van Elmpt, W. (2020) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2020.09.008 · Cited by 326 · see citing works
Improving automatic delineation for head and neck organs at risk by Deep Learning Contouring
van Dijk, L. V., den Bosch, L. V., Aljabar, P., Peressutti, D., Both, S., Steenbakkers, R. J. H. M., Langendijk, J. A., Gooding, M. J., & Brouwer, C. L. (2019) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2019.09.022 · Cited by 240 · see citing works
Self-citationSherer, M. V., Lin, D., Elguindi, S., Duke, S. L., Tan, L. T., Cacicedo, J., Dahele, M. R., & Gillespie, E. F. (2021) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2021.05.003 · Cited by 239 · see citing works
Understanding metric-related pitfalls in image analysis validation
Reinke, A., Tizabi, M. D., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A. E., Rädsch, T., Sudre, C. H., Acion, L., Antonelli, M., et al. (2024) · Nature Methods · doi:10.1038/s41592-023-02150-0 · Cited by 241 · see citing works
Vaassen, F., Hazelaar, C., Vaniqui, A., Gooding, M. J., van der Heyden, B., Canters, R. A. M., & van Elmpt, W. (2019) · Physics and Imaging in Radiation Oncology · doi:10.1016/j.phro.2019.12.001 · Cited by 206 · see citing works
Self-citationApplications and limitations of machine learning in radiation oncology
Jarrett, D. P., Stride, E., Vallis, K. A., & Gooding, M. J. (2019) · British Journal of Radiology · doi:10.1259/bjr.20190001 · Cited by 164 · see citing works
Self-citationMahadevaiah, G., RV, P., Bermejo, I., Jaffray, D., Dekker, A., & Wee, L. (2020) · Medical Physics · doi:10.1002/mp.13562 · Cited by 114 · see citing works
Self-citationAutomated Contouring and Planning in Radiation Therapy: What Is ‘Clinically Acceptable’?
Baroudi, H., Brock, K. K., Cao, W., Chen, X., Chung, C. D., Court, L. E., Basha, M. D. E., Farhat, M., Gay, S., Gronberg, M., et al. (2023) · Diagnostics · doi:10.3390/diagnostics13040667 · Cited by 112 · see citing works
Clinical implementation of deep learning contour autosegmentation for prostate radiotherapy
Cha, E., Elguindi, S., Onochie, I., Gorovets, D. J., Deasy, J. O., Zeléfsky, M. J., & Gillespie, E. F. (2021) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2021.02.040 · Cited by 108 · see citing works
Doolan, P., Charalambous, S., Roussakis, Y., Leczynski, A., Peratikou, M., Benjamin, M., Ferentinos, K., Strouthos, I., Zamboglou, C., & Karagiannis, E. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1213068 · Cited by 108 · see citing works
Harnessing progress in radiotherapy for global cancer control
Jaffray, D. A., Knaul, F. M., Baumann, M. A., & Gospodarowicz, M. K. (2023) · Nature Cancer · doi:10.1038/s43018-023-00619-7 · Cited by 95 · see citing works
A Review of the Metrics Used to Assess Auto-Contouring Systems in Radiotherapy
Mackay, K., Bernstein, D., Glocker, B., Kamnitsas, K., & Taylor, A. (2023) · Clinical Oncology · doi:10.1016/j.clon.2023.01.016 · Cited by 86 · see citing works
Quality Assurance for AI-Based Applications in Radiation Therapy
Claessens, M., Oria, C. S., Brouwer, C. L., Ziemer, B. P., Scholey, J., Lin, H., Witztum, A., Morin, O., Naqa, I. E., van Elmpt, W., et al. (2022) · Seminars in Radiation Oncology · doi:10.1016/j.semradonc.2022.06.011 · Cited by 86 · see citing works
Chamunyonga, C., Edwards, C. R., Caldwell, P. J., Rutledge, P., & Burbery, J. (2020) · Journal of medical imaging and radiation sciences · doi:10.1016/j.jmir.2020.01.008 · Cited by 63 · see citing works
Willigenburg, T., Keizer, D. M. D. M., Peters, M., Claes, A., Lagendijk, J. J. W., de Boer, H. C. J., & van Zyp, J. R. N. V. D. V. (2021) · Clinical and Translational Radiation Oncology · doi:10.1016/j.ctro.2021.01.002 · Cited by 61 · see citing works
Wahid, K. A., Ahmed, S., He, R., van Dijk, L. V., Teuwen, J., McDonald, B. A., Salama, V., Mohamed, A., Salzillo, T. C., Dede, C., et al. (2021) · Clinical and Translational Radiation Oncology · doi:10.1016/j.ctro.2021.10.003 · Cited by 55 · see citing works
External validation of deep learning-based contouring of head and neck organs at risk
Brunenberg, E. J. L., Steinseifer, I. K., van den Bosch, S., Kaanders, J. H. A. M., Brouwer, C. L., Gooding, M. J., van Elmpt, W., & Monshouwer, R. (2020) · Physics and Imaging in Radiation Oncology · doi:10.1016/j.phro.2020.06.006 · Cited by 51 · see citing works
Self-citationRong, Y., Chen, Q., Fu, Y., Yang, X., Al‐Hallaq, H., Wu, Q., Yuan, L., Xiao, Y., Cai, B., Latifi, K., et al. (2023) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2023.10.033 · Cited by 51 · see citing works
Kiser, K. J., Barman, A., Stieb, S., Fuller, C. D., & Giancardo, L. (2021) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-021-00460-3 · Cited by 50 · see citing works
Looney, P., Yin, Y., Collins, S. L., Nicolaides, K. H., Plasencia, W., Molloholli, M., Natsis, S., & Stevenson, G. N. (2021) · IEEE Transactions on Ultrasonics, Ferroelectrics, and Frequency Control · doi:10.1109/tuffc.2021.3052143 · Cited by 40 · see citing works
The predictive value of segmentation metrics on dosimetry in organs at risk of the brain
Poel, R., Rüfenacht, E., Herrmann, E., Scheib, S. G., Manser, P., Aebersold, D. M., & Reyes, M. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102161 · Cited by 39 · see citing works
Heilemann, G., Buschmann, M., Lechner, W., Dick, V., Eckert, F., Heilmann, M., Herrmann, H. O., Moll, M., Knoth, J., Konrad, S., et al. (2023) · Physics and Imaging in Radiation Oncology · doi:10.1016/j.phro.2023.100515 · Cited by 38 · see citing works
Palazzo, G., Mangili, P., Deantoni, C. L., Fodor, A., Broggi, S., Castriconi, R., Ubeira-Gabellini, M. G., Vecchio, A. D., Muzio, N. G. D., & Fiorino, C. (2023) · Physics and Imaging in Radiation Oncology · doi:10.1016/j.phro.2023.100501 · Cited by 35 · see citing works
Hoque, S. M. H., Pirrone, G., Matrone, F., Donofrio, A., Fanetti, G., Caroli, A., Rista, R. S., Bortolus, R., Avanzo, M., Drigo, A., et al. (2023) · Cancers · doi:10.3390/cancers15245735 · Cited by 35 · see citing works
Medical Physics and Imaging–A Timely Perspective
Beyer, T., Bailey, D. L., Birk, U. J., Buvat, I., Catana, C., Cheng, Z., Fang, Q., Giove, F., Kuntner, C., Laistler, E., et al. (2021) · Frontiers in Physics · doi:10.3389/fphy.2021.634693 · Cited by 33 · see citing works
Martí‐Bonmatí, L., Koh, D., Riklund, K., Bobowicz, M., Roussakis, Y., Vilanova, J. C., Fütterer, J. J., Rimola, J., Mallol, P., Ribas, G., et al. (2022) · Insights into Imaging · doi:10.1186/s13244-022-01220-9 · Cited by 33 · see citing works
Kim, N., Chun, J., Chang, J. S., Lee, C. G., Keum, K. C., & Kim, J. S. (2021) · Cancers · doi:10.3390/cancers13040702 · Cited by 33 · see citing works
The impact of organ-at-risk contour variations on automatically generated treatment plans for NSCLC
Vaassen, F., Hazelaar, C., Canters, R. A. M., Peeters, S., Petit, S., & van Elmpt, W. (2021) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2021.08.014 · Cited by 31 · see citing works
Self-citationParkinson, C., Matthams, C., Foley, K., & Spezi, E. (2021) · Radiography · doi:10.1016/j.radi.2021.07.012 · Cited by 30 · see citing works
Fransson, S., Tilly, D., & Strand, R. (2022) · Physics and Imaging in Radiation Oncology · doi:10.1016/j.phro.2022.06.001 · Cited by 28 · see citing works
Claessens, M., Vanreusel, V., Kerf, G. D., Mollaert, I., Löfman, F., Gooding, M. J., Brouwer, C. L., Dirix, P. R., & Verellen, D. (2022) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ac6fad · Cited by 25 · see citing works
Self-citationJohnston, N., Rycke, J. D., Lievens, Y. N., Eijkeren, M. V., Aelterman, J., Vandersmissen, E., Ponte, S., & Vanderstraeten, B. (2022) · Physics and Imaging in Radiation Oncology · doi:10.1016/j.phro.2022.07.004 · Cited by 22 · see citing works
An evaluation of MR based deep learning auto-contouring for planning head and neck radiotherapy
Hague, C. J., McPartlin, A. J., Lee, L. W., Hughes, C., Mullan, D., Beasley, W. J., Green, A., Price, G. J., Whitehurst, P., Slevin, N. J., et al. (2021) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2021.02.018 · Cited by 21 · see citing works
Ouyang, H., Meng, F., Liu, J., Song, X., Li, Y., Yuan, Y., Wang, C., Lang, N., Tian, S., Yao, M., et al. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.814667 · Cited by 21 · see citing works
Taku, N., Wahid, K. A., van Dijk, L. V., Sahlsten, J., Jaskari, J., Kaski, K. K., Fuller, C. D., & Naser, M. A. (2022) · Clinical and Translational Radiation Oncology · doi:10.1016/j.ctro.2022.06.007 · Cited by 21 · see citing works
Chung, S. Y., Chang, J. S., & Kim, Y. B. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1119008 · Cited by 26 · see citing works
Huff, D. T., Santoro-Fernandes, V., Chen, S., Chen, M., Kashuk, C., Weisman, A. J., Jeraj, R., & Perk, T. (2023) · Physics in Medicine and Biology · doi:10.1088/1361-6560/acef8f · Cited by 19 · see citing works
RootPainter3D: Interactive‐machine‐learning enables rapid and accurate contouring for radiotherapy
Smith, A. G., Petersen, J., Terrones‐Campos, C., Berthelsen, A. K., Forbes, N., Darkner, S., Specht, L., & Vogelius, I. R. (2021) · Medical Physics · doi:10.1002/mp.15353 · Cited by 18 · see citing works
Self-citationRusanov, B., Ebert, M. A., Sabet, M., Rowshanfarzad, P., Barry, N., Kendrick, J., Alkhatib, Z., Gill, S., Dass, J., Bucknell, N. W., et al. (2025) · Physical and Engineering Sciences in Medicine · doi:10.1007/s13246-024-01513-x · Cited by 20 · see citing works
Bakx, N., Rijkaart, D., van der Sangen, M., Theuws, J. C. M., van der Toorn, P.-P., Verrijssen, A. E., van der Leer, J., Mutsaers, J., van Nunen, T., Reinders, M., et al. (2023) · Technical Innovations & Patient Support in Radiation Oncology · doi:10.1016/j.tipsro.2023.100211 · Cited by 14 · see citing works
Artificial intelligence contouring in radiotherapy for organs-at-risk and lymph node areas
Meyer, C., Huger, S., Bruand, M., Leroy, T., Palisson, J., Retif, P., Sarrade, T., Barateau, A., Renard-Oldrini, S., Jolnerovski, M., et al. (2024) · Radiation Oncology · doi:10.1186/s13014-024-02554-y · Cited by 14 · see citing works
Sources of performance variability in deep learning-based polyp detection
Tran, T. N., Adler, T. J., Yamlahi, A., Christodoulou, E., Godau, P., Reinke, A., Tizabi, M. D., Sauer, P., Persicke, T., Albert, J., et al. (2023) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-023-02936-9 · Cited by 13 · see citing works
An investigation into the risk of population bias in deep learning autocontouring
McQuinlan, Y., Brouwer, C. L., Lin, Z., Gan, Y., Kim, J. S., van Elmpt, W., & Gooding, M. J. (2023) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2023.109747 · Cited by 13 · see citing works
Self-citationArends, S. R. S., Savenije, M. H. F., Eppinga, W. S. C., van der Velden, J. M., van den Berg, C. A. T., & Verhoeff, J. J. C. (2022) · Physics and Imaging in Radiation Oncology · doi:10.1016/j.phro.2022.02.003 · Cited by 13 · see citing works
Poel, R., Rüfenacht, E., Scheib, S. G., Hemmatazad, H., Krcek, R., Tran, S., Romano, E., Rogers, S. J., Stieb, S., Poolakundan, M. R., et al. (2025) · Communications Medicine · doi:10.1038/s43856-025-01048-6 · Cited by 11 · see citing works
van den Oever, L. B., van Veldhuizen, W. A., Cornelissen, L. J., Spoor, D. S., Willems, T. P., Kramer, G. M., Stigter, T., Rook, M., Crijns, A. P. G., Oudkerk, M., et al. (2022) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-021-00573-9 · Cited by 11 · see citing works
van der Pol, L. H. G., Blanck, O., Grehn, M., Blažek, T., Knybel, L., Balgobind, B. V., Verhoeff, J. J. C., Miszczyk, M., Blamek, S., Reichl, S., et al. (2024) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2024.110610 · Cited by 10 · see citing works
Choi, S. H., Park, J. W., Cho, Y., Yang, G., & Yoon, H. I. (2024) · Cancers · doi:10.3390/cancers16213670 · Cited by 9 · see citing works
Silvério, N. F., Wollenberg, W. V. D., Betgen, A., Wiersema, L., Marijnen, C. A. M., Peters, F., van der Heide, U. A., Simões, R., & Janssen, T. M. (2024) · Advances in Radiation Oncology · doi:10.1016/j.adro.2024.101483 · Cited by 9 · see citing works
Novel methodology to assess the effect of contouring variation on treatment outcome
Jenkins, A., Mullen, T. S., Johnson‐Hart, C., Green, A., McWilliam, A., Aznar, M. C., van Herk, M., & Osorio, E. M. V. (2021) · Medical Physics · doi:10.1002/mp.14865 · Cited by 9 · see citing works
Silvério, N. F., Wollenberg, W. V. D., Betgen, A., Wiersema, L., Marijnen, C. A. M., Peters, F. P., van der Heide, U. A., Simões, R., Intven, M., van der Bijl, E., et al. (2024) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2024.110667 · Cited by 8 · see citing works
Artificial Intelligence Potential Impact on Resident Physician Education in Radiation Oncology
Almeida, N. D., Shekher, R., Pepin, A. N., Schrand, T. V., Goulenko, V., Singh, A. K., & Fung-Kee-Fung, S. (2024) · Advances in Radiation Oncology · doi:10.1016/j.adro.2024.101505 · Cited by 8 · see citing works
Vaassen, F., Zegers, C. M. L., Hofstede, D., Wubbels, M., Beurskens, H., Verheesen, L., Canters, R. A. M., Looney, P., Battye, M., Gooding, M. J., et al. (2023) · Physica Medica · doi:10.1016/j.ejmp.2023.103156 · Cited by 7 · see citing works
Self-citationEisenmann, M., Reinke, A., Weru, V., Tizabi, M. D., Isensee, F., Adler, T. J., Ali, S., Andrearczyk, V., Aubreville, M., Baid, U., et al. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01911 · Cited by 7 · see citing works
Wahid, K. A., Cárdenas, C., Marquez, B., Netherton, T., Kann, B. H., Court, L. E., He, R., Naser, M. A., Moreno, A. C., Fuller, C. D., et al. (2024) · Advances in Radiation Oncology · doi:10.1016/j.adro.2024.101521 · Cited by 6 · see citing works
Bürkle, S. L., Kuhn, D., Fechter, T., Radicioni, G., Hartong, N., Freitag, M. T., Qiu, X., Karagiannis, E., Grosu, A., Baltas, D., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-76288-y · Cited by 5 · see citing works
Geometrically focused training and evaluation of organs‐at‐risk segmentation via deep learning
Ni, R., Chuk, E., Han, K. C.-Z., Croke, J. M., Fyles, A., Lukovic, J., Milosevic, M., Haibe‐Kains, B., & Rink, A. (2025) · Medical Physics · doi:10.1002/mp.17840 · Cited by 5 · see citing works
Yeung, C., Ungi, T., Hu, Z., Jamzad, A., Kaufmann, M., Walker, R., Merchant, S. J., Engel, C. J., Jabs, D., Rudan, J. F., et al. (2024) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-024-03133-y · Cited by 5 · see citing works
Weisman, A. J., Huff, D. T., Govindan, R. M., Chen, S., & Perk, T. (2023) · Biomedical Physics & Engineering Express · doi:10.1088/2057-1976/acfb06 · Cited by 5 · see citing works
Wubbels, M., Ribeiro, M., Wolterink, J. M., van Elmpt, W., Compter, I., Hofstede, D., Birimac, N. E., Vaassen, F., Palmgren, K., Hansen, H. H. G., et al. (2025) · Cancers · doi:10.3390/cancers17101598 · Cited by 4 · see citing works
Constantinou, A., Hoole, A. C. F., Wong, D., Sagoo, G. S., Alvarez-Valle, J., Takeda, K., Griffiths, T., Edwards, A., Robinson, A., Stubbington, L., et al. (2024) · Clinical Oncology · doi:10.1016/j.clon.2024.10.032 · Cited by 4 · see citing works
Cornelissen, S., Schouten, S. M., Langenhuizen, P. P. J. H., Kunst, H. P. M., Verheul, J. B., & de With, P. H. N. (2025) · Neuroradiology · doi:10.1007/s00234-025-03611-3 · Cited by 4 · see citing works
Dose-based evaluation of delineation variation in radiotherapy: A scoping review
van Aalst, J. E., Maruccio, F. C., Simões, R., Mody, P., Gouw, Z. A. R., Wijsman, R., Sonke, J., van Ooijen, P. M. A., Brouwer, C. L., & Janssen, T. M. (2026) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2026.111595 · Cited by 3 · see citing works
Mazonakis, M., Tzanis, E., Kachris, S., Lyraraki, E., & Damilakis, J. E. (2025) · Physica Medica · doi:10.1016/j.ejmp.2025.104896 · Cited by 3 · see citing works
On the Evaluation of Auto-Contouring in Radiotherapy
Gooding, M. J. (2021) · CRC Press eBooks · doi:10.1201/9780429323782-18 · Cited by 3 · see citing works
Self-citationRauseo, E., Omer, M., Amir-Khalili, A., Sojoudi, A., Le, T., Cook, S. A., Hausenloy, D. J., Ang, B., Toh, D., Bryant, J., et al. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2021.816985 · Cited by 3 · see citing works
He, D., Udupa, J. K., Tong, Y., & Torigian, D. A. (2023) · SPIE Medical Imaging: Imaging Informatics for Healthcare, Research, and Applications · doi:10.1117/12.2654421 · Cited by 3 · see citing works
Kuijer, K. M., Smits, H. J. G., Doornaert, P., Niu, K., Savenije, M. H. F., Smid, E. J., Terhaard, C. H. J., Terpstra, M. L., de Ridder, M., & Philippens, M. E. P. (2025) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2024.12.009 · Cited by 2 · see citing works
Fransson, S. (2024) · Physics and Imaging in Radiation Oncology · doi:10.1016/j.phro.2024.100551 · Cited by 1 · see citing works
Oliva, P. J., Ghosh, S., Huang, F., Wiebe, E. M., Cuartero, J., Ghosh, S. S., Boulanger, P., Yun, J., Punithakumar, K., & Menon, G. (2026) · Brachytherapy · doi:10.1016/j.brachy.2025.12.007 · Cited by 1 · see citing works
Wahid, K. A., Xu, J., El-Habashy, D. M., Khamis, Y., Abdelaal, M. A., McDonald, B. A., O’Connell, N., Thill, D., Ahmed, S., Sharafi, C. S., et al. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.975902 · Cited by 1 · see citing works
Lee, D. S., Lee, Y. K., Kang, Y. N., Won, Y. K., Park, S. H., Kim, Y. S., Kim, J. S., & Won, H. S. (2021) · Scientific Reports · doi:10.1038/s41598-020-78666-8 · Cited by 1 · see citing works
Karagiannis, E. (2021) · Journal of Oncology Research and Therapy · doi:10.29011/2574-710x.010113 · Cited by 1 · see citing works
Machine learning & semantic web technologies for cancer care
Geetha, M. (2020) · doi:10.26481/dis.20200409gm · Cited by 1 · see citing works
Why is the winner the best?Preprint
Eisenmann, M., Reinke, A., Weru, V., Tizabi, M. D., Isensee, F., Adler, T. J., Ali, S., Andrearczyk, V., Aubreville, M., Baid, U., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.17719 · Cited by 1 · see citing works
Clinical data science in radiotherapy
van Soest, J. (2018) · doi:10.26481/dis.20181128js
Application of Deep Learning in Radiation Therapy
Rawat, S., Singh, S., Alam, M. A., & Malviya, R. (2022) · doi:10.1002/9781119857983.ch10
MRI-guided stereotactic radiotherapy for localised prostate cancer
Willigenburg, T. (2022) · doi:10.33540/1577
Role of MRI in Radiation Oncology
Das, I. J., Yadav, P., Alongi, F., & Mittal, B. B. (2024) · Springer eBooks · doi:10.1007/978-3-031-48165-9_2
Hellman, S. E., Voros, L., Yu, V., Lovelock, D. M., Berry, S. L., Zhang, L., Hunt, M. A., Deasy, J. O., & Cerviño, L. (2024) · Advances in Radiation Oncology · doi:10.1016/j.adro.2024.101544
Cant, I., Sykes, J., Surjan, Y., Beldham‐Collins, R., Murphy, L., Fernandez, J., & Salkeld, A. (2025) · Journal of Medical Radiation Sciences · doi:10.1002/jmrs.893
Evaluation of Two Commercial Artificial Intelligence Segmentation Systems for Radiation Therapy
DiTusa, C., Chen, J., Husain, A., Schneider, C., Spears, H., Nikos Paragios, A., Debevec, G., & Stathakis, S. (2025) · Journal of Medical Physics · doi:10.4103/jmp.jmp_11_25
Zhang, W. (2026) · Current Radiopharmaceuticals · doi:10.1016/j.craph.2026.100037
Gupta, V., Sharma, M., Chowdhary, R. L., Aggarwal, S., Lal, M., Murali, M., Singamsetty, M., Chaudhari, S., Chowdhary, M. G., & Beriwal, S. (2026) · Frontiers in Oncology · doi:10.3389/fonc.2026.1809864
Evaluating artificial intelligence versus human-generated contours in the thorax
Icht, O., Farkas, P., Liu, M., Marants, R., Santiago, A. T., Cho, B. C. J., Lok, B. H., Sun, A. Y., Mesci, A., Wong, H., et al. (2026) · Physics and Imaging in Radiation Oncology · doi:10.1016/j.phro.2026.101037
Coc, F. L., & Marcu, L. G. (2026) · Physica Medica · doi:10.1016/j.ejmp.2026.105901
Common Limitations of Image Processing Metrics: A Picture Story
Reinke, A., Tizabi, M. D., Sudre, C. H., Eisenmann, M., Rädsch, T., Baumgartner, M., Acion, L., Antonelli, M., Arbel, T., Bakas, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2104.05642 · Cited by 126 · see citing works
Metrics reloaded: Recommendations for image analysis validation
Maier-Hein, L., Reinke, A., Godau, P., Tizabi, M. D., Buettner, F., Christodoulou, E., Glocker, B., Isensee, F., Kleesiek, J., Kozubek, M., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.01653 · Cited by 59 · see citing works
- Authors
- Gooding, M. J., Smith, A. J., Tariq, M., Aljabar, P., Peressutti, D., van der Stoep, J., Reymen, B., Emans, D., Hattu, D., van Loon, J., de Rooy, M., Wanders, R., Peeters, S., Lustberg, T., van Soest, J., Dekker, A., & van Elmpt, W.
- Affiliations
- Mirada Medical (United Kingdom) (GB)
- Department of Radiation Oncology (MAASTRO) GROW School for Oncology and Developmental Biology Maastricht University Medical Centre+ Dr Tanslaan 12 6229ET Maastricht The Netherlands
- Maastricht University Medical Centre (NL)
- Corresponding author
- Gooding, Mark J.
- ORCID
- Venue
- Medical Physics
- Publisher
- Wiley
- Published
- 2018-10-12
- Volume
- 45
- Issue
- 11
- Pages
- 5105-5115
- ISSN
- 0094-2405 (ISSN-L)
- 2473-4209
- OpenAlex
- W2890012362
- Cited by
- 89
- References
- 21
- Citations by year
- 2026: 6 · 2025: 9 · 2024: 15 · 2023: 18 · 2022: 14 · 2021: 18 · 2020: 5 · 2019: 3 · 2018: 1
- FWCI
- 3.38
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Medical Physics (publishedVersion)
- PubMed (publishedVersion)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Funders
- Innovate UK (600277)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Physics and Astronomy › Radiation › Advanced Radiotherapy Techniques
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- Turing test, thoracic organs, radiation oncology, Dice coefficient, esophagus, heart, lung, spinal cord, radiotherapy treatment planning
- MeSH terms
- Machine Learning, Carcinoma, Non-Small-Cell Lung, Humans, Image Processing, Computer-Assisted, Lung Neoplasms, Tomography, X-Ray Computed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
