MRIQC: Advancing the automatic prediction of image quality in MRI from unseen sites
Open accessPublished version of a preprint
Abstract
Quality control of MRI is essential for excluding problematic acquisitions and avoiding bias in subsequent image processing and analysis. Visual inspection is subjective and impractical for large scale datasets. Although automated quality assessments have been demonstrated on single-site datasets, it is unclear that solutions can generalize to unseen data acquired at new sites. Here, we introduce the MRI Quality Control tool (MRIQC), a tool for extracting quality measures and fitting a binary (accept/exclude) classifier. Our tool can be run both locally and as a free online service via the OpenNeuro.org portal. The classifier is trained on a publicly available, multi-site dataset (17 sites, N = 1102). We perform model selection evaluating different normalization and feature exclusion approaches aimed at maximizing across-site generalization and estimate an accuracy of 76%±13% on new sites, using leave-one-site-out cross-validation. We confirm that result on a held-out dataset (2 sites, N = 265) also obtaining a 76% accuracy. Even though the performance of the trained classifier is statistically above chance, we show that it is susceptible to site effects and unable to account for artifacts specific to new sites. MRIQC performs with high accuracy in intra-site prediction, but performance on unseen sites leaves space for improvement which might require more labeled data and new approaches to the between-site variability. Overcoming these limitations is crucial for a more objective quality assessment of neuroimaging data, and to enable the analysis of extremely large and multi-site samples.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The average temporal signal‐to‐noise ratio was 50.4, as assessed by the MRIQC toolbox.
“The average temporal signal‐to‐noise ratio was 50.4, as assessed by the MRIQC toolbox 69.”
From History of early life adversity is associated with increased food addiction and sex-specific alterations in reward network connectivity in obesity (2019) · cites this paper as [69]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · see citing works
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016 · see citing works
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421 · see citing works
Cortes, C., & Vapnik, V. N. (1995) · Machine Learning · doi:10.1023/a:1022627411411 · Cited by 34,565 · see citing works
Jenkinson, M., Beckmann, C. F., Behrens, T. E. J., Woolrich, M. W., & Smith, S. M. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.09.015 · Cited by 11,314 · see citing works
Fischl, B. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.01.021 · Cited by 9,526 · see citing works
Geurts, P., Ernst, D., & Wehenkel, L. (2006) · Machine Learning · doi:10.1007/s10994-006-6226-1 · Cited by 9,300 · see citing works
Power, J. D., Barnes, K. A., Snyder, A. Z., Schlaggar, B. L., & Petersen, S. E. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.10.018 · Cited by 7,429 · see citing works
(2005) · PubMed · Cited by 7,011 · see citing works
Practical Bayesian Optimization of Machine Learning Algorithms
Snoek, J., Larochelle, H., & Adams, R. P. (2012) · arXiv · doi:10.48550/arxiv.1206.2944 · Cited by 6,691 · see citing works
Pattern Recognition and Neural Networks
Ripley, B. D. (1996) · Cambridge University Press eBooks · doi:10.1017/cbo9780511812651 · Cited by 6,446 · see citing works
Di Martino, A., Yan, C.-G., Li, Q., Denio, E., Castellanos, F. X., Alaerts, K., Anderson, J. S., Assaf, M., Bookheimer, S. Y., Dapretto, M., et al. (2014) · Molecular Psychiatry · doi:10.1038/mp.2013.78 · Cited by 2,780 · see citing works
Singularity: Scientific containers for mobility of compute
Kurtzer, G. M., Sochat, V., & Bauer, M. W. (2017) · PLOS ONE · doi:10.1371/journal.pone.0177459 · Cited by 2,740 · see citing works
The Human Connectome Project: A data acquisition perspective
Essen, D. C. V., Uǧurbil, K., Auerbach, E. J., Barch, D. M., Behrens, T. E. J., Bucholz, R. D., Chang, A., Chen, L., Corbetta, M., Curtiss, S. W., et al. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.02.018 · Cited by 2,605 · see citing works
Nipype: A Flexible, Lightweight and Extensible Neuroimaging Data Processing Framework in Python
Gorgolewski, K. J., Burns, C. D., Madison, C. M., Clark, D., Halchenko, Y. O., Waskom, M., & Ghosh, S. (2011) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2011.00013 · Cited by 2,206 · see citing works
Self-citationTackling the widespread and critical impact of batch effects in high-throughput data
Leek, J. T., Scharpf, R. B., Bravo, H. C., Simcha, D., Langmead, B., Johnson, W. E., Geman, D., Baggerly, K., & Irizarry, R. A. (2010) · Nature Reviews Genetics · doi:10.1038/nrg2825 · Cited by 2,196 · see citing works
Unbiased nonlinear average age-appropriate brain templates from birth to adulthood
Fonov, V., Evans, A., McKinstry, R. C., Almli, C. R., & Collins, D. L. (2009) · NeuroImage · doi:10.1016/s1053-8119(09)70884-5 · Cited by 2,092 · see citing works
Multimodal population brain imaging in the UK Biobank prospective epidemiological study
Miller, K. L., Alfaro-Almagro, F., Bangerter, N. K., Thomas, D. L., Yacoub, E., Xu, J., Bartsch, A. J., Jbabdi, S., Sotiropoulos, S. N., Andersson, J. L. R., et al. (2016) · Nature Neuroscience · doi:10.1038/nn.4393 · Cited by 2,020 · see citing works
Gorgolewski, K. J., Auer, T., Calhoun, V. D., Craddock, R. C., Das, S., Duff, E. P., Flandin, G., Ghosh, S. S., Glatard, T., Halchenko, Y. O., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.44 · Cited by 1,775 · see citing works
Self-citationHarmonization of cortical thickness measurements across scanners and sites
Fortin, J.-P., Cullen, N., Sheline, Y. I., Taylor, W. D., Aselcioglu, I., Cook, P. A., Adams, P., Cooper, C., Fava, M., McGrath, P. J., et al. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.11.024 · Cited by 1,580 · see citing works
Dietrich, O., Raya, J. G., Reeder, S. B., Reiser, M. F., & Schoenberg, S. O. (2007) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.20969 · Cited by 1,001 · see citing works
Pattern Recognition and Neural Networks
Fulkerson, B., & Ripley, B. D. (1997) · Technometrics · doi:10.2307/1270922 · Cited by 830 · see citing works
Deriving reproducible biomarkers from multi-site resting-state data: An Autism-based example
Abraham, A., Milham, M. P., Di Martino, A., Craddock, R. C., Samaras, D., Thirion, B., & Varoquaux, G. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.10.045 · Cited by 701 · see citing works
Scheffer, T. (2005) · IGI Global eBooks · doi:10.4018/978-1-59140-557-3.ch192 · Cited by 629 · see citing works
Permutation Tests for Studying Classifier Performance
Ojala, M., & Garriga, G. C. (2009) · Proceedings · doi:10.1109/icdm.2009.108 · Cited by 578 · see citing works
Spurious group differences due to head motion in a diffusion MRI study
Yendiki, A., Koldewyn, K., Kakunoori, S., Kanwisher, N., & Fischl, B. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.11.027 · Cited by 508 · see citing works
Head motion during MRI acquisition reduces gray matter volume and thickness estimates
Reuter, M., Tisdall, M. D., Qureshi, A., Buckner, R. L., van der Kouwe, A. J. W., & Fischl, B. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2014.12.006 · Cited by 457 · see citing works
A phenome-wide examination of neural and cognitive function
Poldrack, R. A., Congdon, E. L., Triplett, W. T., Gorgolewski, K. J., Karlsgodt, K. H., Mumford, J. A., Sabb, F. W., Freimer, N. B., London, E. D., Cannon, T. D., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.110 · Cited by 414 · see citing works
Self-citationMeasuring signal-to-noise ratios in MR imaging.
Kaufman, L., Kramer, D., Crooks, L. E., et al. (1989) · Radiology · doi:10.1148/radiology.173.1.2781018 · Cited by 361 · see citing works
Permutation Tests for Studying Classifier Performance
Ojala, M., & Garriga, G. C. (2010) · Journal of Machine Learning Research · doi:10.5555/1756006.1859913 · Cited by 343 · see citing works
Toward open sharing of task-based fMRI data: the OpenfMRI project
Poldrack, R. A., Barch, D. M., Mitchell, J. P., Wager, T. D., Wagner, A. D., Devlin, J. T., Cumba, C., Koyejo, S., & Milham, M. P. (2013) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2013.00012 · Cited by 341 · see citing works
Self-citationGorgolewski, K. J., Alfaro-Almagro, F., Auer, T., Bellec, P., Capotă, M., Chakravarty, M. M., Churchill, N. W., Cohen, A. L., Craddock, R. C., Devenyi, G. A., et al. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005209 · Cited by 311 · see citing works
Self-citationPrice, R. R., Axel, L., Morgan, T. J., et al. (1990) · Medical Physics · doi:10.1118/1.596566 · Cited by 308 · see citing works
Atkinson, D., Hill, D., Stoyle, P. N. R., Summers, P. E., & Keevil, S. (1997) · IEEE Transactions on Medical Imaging · doi:10.1109/42.650886 · Cited by 282 · see citing works
Test–retest and between‐site reliability in a multicenter fMRI study
Friedman, L., Stern, H. S., Brown, G. G., Mathalon, D. H., Turner, J. A., Glover, G. H., Gollub, R. L., Lauriello, J., Lim, K. O., Cannon, T. D., et al. (2007) · Human Brain Mapping · doi:10.1002/hbm.20440 · Cited by 245 · see citing works
Measurement of Signal-to-Noise and Contrast-to-Noise in the fBIRN Multicenter Imaging Study
Magnotta, V. A., Friedman, L., & BIRN, F. (2006) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-006-0264-x · Cited by 194 · see citing works
Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm
Littlestone, N. (1987) · Annual Symposium on Foundations of Computer Science · doi:10.1109/sfcs.1987.37 · Cited by 186 · see citing works
Alexander-Bloch, A., Clasen, L., Stockman, M., Ronan, L., Lalonde, F., Giedd, J., & Raznahan, A. (2016) · Human Brain Mapping · doi:10.1002/hbm.23180 · Cited by 178 · see citing works
Automatic quality assessment in structural brain magnetic resonance imaging
Mortamet, B., Bernstein, M. A., Jack, C. R., Gunter, J. L., Ward, C. P., Britson, P. J., Meuli, R. A., Thiran, J., & Krüger, G. (2009) · Magnetic Resonance in Medicine · doi:10.1002/mrm.21992 · Cited by 171 · see citing works
Backhausen, L. L., Herting, M. M., Buse, J., Roessner, V., Smolka, M. N., & Vetter, N. C. (2016) · Frontiers in Neuroscience · doi:10.3389/fnins.2016.00558 · Cited by 147 · see citing works
Harmonization of cortical thickness measurements across scanners and sitesPreprint
Fortin, J., Cullen, N. C., Sheline, Y. I., et al. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/148502 · Cited by 88 · see citing works
No-Reference Image Quality Metrics for Structural MRI
Woodard, J. P., & Carley-Spencer, M. P. (2006) · Neuroinformatics · doi:10.1385/ni:4:3:243 · Cited by 87 · see citing works
Pizarro, R., Cheng, X., Barnett, A. S., Lemaître, H., Verchinski, B. A., Goldman, A. L., Xiao, E., Luo, Q., Berman, K. F., Callicott, J. H., et al. (2016) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2016.00052 · Cited by 73 · see citing works
OpenNeuro – a free online platform for sharing and analysis of neuroimaging data
Gorgolewski, K. J., Estéban, O., Schaefer, G., Wandell, B. A., & Poldrack, R. A. (2017) · doi:10.7490/f1000research.1114354.1 · Cited by 71 · see citing works
Self-citationGanzetti, M., Wenderoth, N., & Mantini, D. (2016) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2016.00010 · Cited by 70 · see citing works
Automated quality control of brain MR images
Gedamu, E. L., Collins, D. L., & Arnold, D. L. (2008) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.21434 · Cited by 59 · see citing works
Zarrar, S., Giavasis, S., Li, Q., Benhajali, Y., Chaogan, Y., Yang, Z., Milham, M. P., Bellec, P., & Cameron, C. (2015) · Frontiers in Neuroscience · doi:10.3389/conf.fnins.2015.91.00047 · Cited by 59 · see citing works
Gardner, E. A., Ellis, J. H., Hyde, R. J., et al. (1995) · Academic Radiology · doi:10.1016/s1076-6332(05)80184-9 · Cited by 24 · see citing works
Organization for Human Brain Mapping
(2015) · F1000Research Channels · doi:10.12688/f1000research.channels.58 · Cited by 14 · see citing works
Organization for Human Brain Mapping
(2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.04.015 · Cited by 12 · see citing works
Biswal (1997) · doi:10.1002/(sici)1099-1492(199706/08)10:4/5
Flagged first, then by citations
Cited by
Sorted by citations
fMRIPrep: A robust preprocessing pipeline for functional MRI
Esteban, O., Markiewicz, C. J., Blair, R. W., Moodie, C. A., Isik, A. I., Erramuzpe, A., Kent, J. D., Goncalves, M., DuPre, E., Snyder, M., et al. (2019) · Nature Methods · doi:10.1038/s41592-018-0235-4 · Cited by 3,905 · see citing works
Self-citationAnalysis of task-based functional MRI data preprocessed with fMRIPrep
Esteban, O., Ciric, R., Finc, K., Blair, R. W., Markiewicz, C. J., Moodie, C. A., Kent, J. D., Goncalves, M., DuPre, E., Gomez, D. E. P., et al. (2020) · Nature Protocols · doi:10.1038/s41596-020-0327-3 · Cited by 315 · see citing works
Self-citationAcute social isolation evokes midbrain craving responses similar to hunger
Tomova, L., Wang, K. L., Thompson, T., Matthews, G. A., Takahashi, A., Tye, K. M., & Saxe, R. (2020) · Nature Neuroscience · doi:10.1038/s41593-020-00742-z · Cited by 283 · see citing works
White, T., Blok, E., & Calhoun, V. D. (2022) · Human Brain Mapping · doi:10.1002/hbm.25120 · Cited by 176 · see citing works
TAPAS: An Open-Source Software Package for Translational Neuromodeling and Computational Psychiatry
Frässle, S., Aponte, E. A., Bollmann, S., Brodersen, K. H., Do, C. T., Harrison, O. K., Harrison, S. J., Heinzle, J., Iglesias, S., Kasper, L., et al. (2021) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2021.680811 · Cited by 169 · see citing works
Brain ventricular volume changes induced by long-duration spaceflight
Van Ombergen, A., Jillings, S., Jeurissen, B., Tomilovskaya, E., Rumshiskaya, A., Litvinova, L., Nosikova, I., Pechenkova, E., Rukavishnikov, I., Manko, O., et al. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1820354116 · Cited by 148 · see citing works
Cogitate Consortium, Ferrante, O., Gorska-Klimowska, U., Henin, S., Hirschhorn, R., Khalaf, A., Lepauvre, A., Liu, L., Richter, D., Vidal, Y., et al. (2025) · Nature · doi:10.1038/s41586-025-08888-1 · Cited by 136 · see citing works
Judd, N., Sauce, B., Wiedenhoeft, J., Tromp, J., Chaarani, B., Schliep, A., van Noort, B., Penttilä, J., Grimmer, Y., Insensee, C., et al. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2001228117 · Cited by 105 · see citing works
Eklund, A., Knutsson, H., & Nichols, T. E. (2019) · Human Brain Mapping · doi:10.1002/hbm.24350 · Cited by 82 · see citing works
Educational disparities in brain health and dementia across Latin America and the United States
Gonzalez‐Gomez, R., Legaz, A., Moguilner, S., Cruzat, J., Hernández, H., Baez, S., Cocchi, R., Coronel‐Olivero, C., Medel, V., Tagliazuchi, E., et al. (2024) · Alzheimer's & Dementia · doi:10.1002/alz.14085 · Cited by 55 · see citing works
Technical Note: MRQy — An open‐source tool for quality control of MR imaging data
Sadri, A. R., Janowczyk, A., Zhou, R., Verma, R., Beig, N., Antunes, J., Madabhushi, A., Tiwari, P., & Viswanath, S. E. (2020) · Medical Physics · doi:10.1002/mp.14593 · Cited by 63 · see citing works
A collaborative resource platform for non-human primate neuroimaging
Messinger, A., Sirmpilatze, N., Heuer, K., Loh, K. K., Mars, R. B., Sein, J., Xu, T., Glen, D., Jung, B., Seidlitz, J., et al. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2020.117519 · Cited by 52 · see citing works
Osadchiy, V., Mayer, E. A., Bhatt, R., Labus, J. S., Gao, L., Kilpatrick, L. A., Liu, C., Tillisch, K., Naliboff, B., Chang, L., et al. (2019) · Obesity Science & Practice · doi:10.1002/osp4.362 · Cited by 44 · see citing works
Thomas, G. E. C., Zeidman, P., Sultana, T., Zarkali, A., Razi, A., & Weil, R. S. (2022) · Brain Communications · doi:10.1093/braincomms/fcac329 · Cited by 38 · see citing works
Reynés-Llompart, G., Sabaté-Llobera, A., Llinares-Tello, E., Martí-Climent, J. M., & Gámez-Cenzano, C. (2019) · Scientific Reports · doi:10.1038/s41598-019-46937-8 · Cited by 35 · see citing works
Gonzalez-Gomez, R., Cruzat, J., Hernández, H., Migeot, J., Legaz, A., Santamaria-García, H., Fittipaldi, S., Maito, M. A., Medel, V., Tagliazucchi, E., et al. (2025) · eClinicalMedicine · doi:10.1016/j.eclinm.2025.103187 · Cited by 19 · see citing works
Yazdan-Panah, A., Stankoff, B., & Colliot, O. (2024) · Medical Imaging 2024: Image Processing · doi:10.1117/12.3006580 · Cited by 1 · see citing works
- Authors
- Esteban, O., Birman, D., Schaer, M., Koyejo, O. O., Poldrack, R. A., & Gorgolewski, K. J.
- Affiliations
- Stanford University (US)
- University of Geneva (CH)
- ORCID
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-09-25
- Volume
- 12
- Issue
- 9
- Pages
- e0184661
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC5612458
- OpenAlex
- W2950120067
- Cited by
- 920
- References
- 51
- Citations by year
- 2026: 109 · 2025: 157 · 2024: 141 · 2023: 141 · 2022: 127 · 2021: 105 · 2020: 72 · 2019: 45 · 2018: 22 · 2017: 1
- FWCI
- 29.36
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Laura and John Arnold Foundation (US)
- Swiss National Science Foundation (SNSF) (158831,163859)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced MRI Techniques and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Keywords
- MRIQC, MRI quality control, automated image quality assessment, OpenNeuro, feature normalization
- MeSH terms
- Brain, Humans, Image Enhancement, Image Interpretation, Computer-Assisted, Magnetic Resonance Imaging, Software, Observer Variation, Neuroimaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
