Machine Learning with Multi-Site Imaging Data: An Empirical Study on the Impact of Scanner Effects
Open accessAbstract
This is an empirical study to investigate the impact of scanner effects when using machine learning on multi-site neuroimaging data. We utilize structural T1-weighted brain MRI obtained from two different studies, Cam-CAN and UK Biobank. For the purpose of our investigation, we construct a dataset consisting of brain scans from 592 age- and sex-matched individuals, 296 subjects from each original study. Our results demonstrate that even after careful pre-processing with state-of-the-art neuroimaging pipelines a classifier can easily distinguish between the origin of the data with very high accuracy. Our analysis on the example application of sex classification suggests that current approaches to harmonize data are unable to remove scanner-specific bias leading to overly optimistic performance estimates and poor generalization. We conclude that multi-site data harmonization remains an open challenge and particular care needs to be taken when using such data with advanced machine learning methods for predictive modelling.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Appropriately investigating variety of data sources can provide a strong indication for the applicability and generalisability of the ML application in different environments.
“Appropriately investigating such questions can provide a strong indication for the applicability and generalisability of the ML application in different environments220–223.”
From The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review (2024) · cites this paper as [223]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Sudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., Downey, P., Elliott, P., Green, J., Landray, M., et al. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001779 · Cited by 12,728 · see citing works
Zhang, Y., Brady, M., & Smith, S. M. (2001) · IEEE Transactions on Medical Imaging · doi:10.1109/42.906424 · Cited by 7,628 · see citing works
N4ITK: Improved N3 Bias Correction
Tustison, N. J., Avants, B., Cook, P. A., Zheng, Y., Egan, A., Yushkevich, P. A., & Gee, J. C. (2010) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2010.2046908 · Cited by 6,119 · see citing works
Multimodal population brain imaging in the UK Biobank prospective epidemiological study
Miller, K. L., Alfaro-Almagro, F., Bangerter, N. K., Thomas, D. L., Yacoub, E., Xu, J., Bartsch, A. J., Jbabdi, S., Sotiropoulos, S. N., Andersson, J. L. R., et al. (2016) · Nature Neuroscience · doi:10.1038/nn.4393 · Cited by 2,020 · see citing works
Harmonization of cortical thickness measurements across scanners and sites
Fortin, J.-P., Cullen, N., Sheline, Y. I., Taylor, W. D., Aselcioglu, I., Cook, P. A., Adams, P., Cooper, C., Fava, M., McGrath, P. J., et al. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.11.024 · Cited by 1,580 · see citing works
Image processing and Quality Control for the first 10,000 brain imaging datasets from UK Biobank
Alfaro‐Almagro, F., Jenkinson, M., Bangerter, N. K., Andersson, J., Griffanti, L., Douaud, G., Sotiropoulos, S. N., Jbabdi, S., Hernández-Fernández, M., Vallée, E., et al. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.10.034 · Cited by 1,563 · see citing works
Harmonization of multi-site diffusion tensor imaging data
Fortin, J.-P., Parker, D., Tunç, B., Watanabe, T., Elliott, M. A., Ruparel, K., Roalf, D. R., Satterthwaite, T. D., Gur, R. C., Gur, R. E., et al. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.08.047 · Cited by 1,261 · see citing works
New variants of a method of MRI scale standardization
Nyúl, L. G., Udupa, J. K., & Zhang, X. (2000) · IEEE Transactions on Medical Imaging · doi:10.1109/42.836373 · Cited by 958 · see citing works
Taylor, J. R., Williams, N., Cusack, R., Auer, T., Shafto, M. A., Dixon, M., TYLER, L. K., Cam‐CAN, & Henson, R. N. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.09.018 · Cited by 759 · see citing works
Robust Brain Extraction Across Datasets and Comparison With Publicly Available Methods
Iglesias, J. E., Liu, C., Thompson, P. M., & Tu, Z. (2011) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2011.2138152 · Cited by 561 · see citing works
Ashburner, J. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.10.025 · Cited by 522 · see citing works
Statistical Challenges in “Big Data” Human Neuroimaging
Smith, S. M., & Nichols, T. E. (2018) · Neuron · doi:10.1016/j.neuron.2017.12.018 · Cited by 331 · see citing works
Quantifying Confounding Bias in Neuroimaging Datasets with Causal Inference
Wachinger, C., Becker, B. G., Rieckmann, A., & Pölsterl, S. (2019) · arXiv · doi:10.48550/arxiv.1907.04102
Flagged first, then by citations
Research works printed but not linked: 2
- Friston, K., Ashburner, J., Kiebel, S., Nichols, T., and Penny, W., editors (2007). Statistical Parametric Mapping: The Analysis of Functional Brain Images . Academic Press.
- Shafto, M. A., Tyler, L. K., Dixon, M., Taylor, J. R., Rowe, J. B., Cusack, R., Calder, A. J., Marslen-Wilson, W. D., Duncan, J., Dalgleish, T., et al. (2014). The Cambridge Centre for Ageing and Neuroscience (Cam-CAN) study protocol: a cross-sectional, lifespan, multidisciplinary examination of healthy cognitive ageing. BMC Neurology , 14(1):204.
Cited by
Sorted by citations
The Liver Tumor Segmentation Benchmark (LiTS)
Bilic, P., Christ, P., Li, H. B., Vorontsov, E., Ben-Cohen, A., Kaissis, G., Szeskin, A., Jacobs, C., Mamani, G. E. H., Chartrand, G., et al. (2023) · Medical Image Analysis · doi:10.1016/j.media.2022.102680 · Cited by 1,412 · see citing works
Self-citationCausality matters in medical imaging
Castro, D. C., Walker, I., & Glocker, B. (2020) · Nature Communications · doi:10.1038/s41467-020-17478-w · Cited by 364 · see citing works
Self-citationDeep learning-based unlearning of dataset bias for mri harmonisation and confound removal
Dinsdale, N. K., Jenkinson, M., & Namburete, A. I. L. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2020.117689 · Cited by 178 · see citing works
The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review
Schwabe, D., Becker, K., Seyferth, M., Klaß, A., & Schaeffter, T. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01196-4 · Cited by 169 · see citing works
- Authors
- Glocker, B., Robinson, R., Castro, D. C., Dou, Q., & Konukoglu, E.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1910.04597
- OpenAlex
- W2979557864
- Cited by
- 74
- References
- 13
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 17 · 2023: 10 · 2022: 8 · 2021: 21 · 2020: 9 · 2019: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion, other-oa)
- arXiv (Cornell University) (cc-by)
- Funders
- European Commission (ERC-2017, 757173, HORIZON2020)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced MRI Techniques and Applications
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced Neuroimaging Techniques and Applications
- Keywords
- scanner effects, data harmonization, structural MRI, sex classification, classifier generalization, overfitting, Cam-CAN, UK Biobank
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
