A comparison of Freesurfer and multi-atlas MUSE for brain anatomy segmentation: Findings about size and age bias, and inter-scanner stability in multi-site aging studies

Probably not a substantial claim: may not pass S3 (load-bearing).

MUSE has been used in several studies.

“and has been used in several studies (Habes et al., 2016; Satterthwaite et al., 2016; Wee 2017; Tian et al., 2018).”

• cites Habes et al., 2016 sourceSupported
• cites Satterthwaite et al., 2016 sourceDisputed

  The Methods (Image processing; page unavailable) identify the segmentation procedure used in this study as MARS, not MUSE. Thus this paper does not substantiate its inclusion as an example of a study using MUSE. (LLM, conf=0.82)

  Checked by a reasoning model — may be wrong.

  Experimental — accuracy not yet measured
• cites Wee 2017 sourceCould not verify

  The cited paper’s Signal Transduction section discusses EGFR signaling and cell proliferation, not use of MUSE. Searches for “MUSE” and its neuroimaging-segmentation context returned no relevant passage in our copy, so this source does not verify the attributed claim. (LLM, conf=0.77)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

An extensive body of literature has relied on Freesurfer segmentations.

“At present, an important body of literature and neuropsychiatry findings are based on Freesurfer segmentations (Kikinis et al., 2010; Rohrer2011; Messina et al., 2011; Sabuncu et al., 2011).”

• cites Kikinis et al., 2010 sourceSupported
• cites Rohrer2011Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Messina et al., 2011 sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Consensus segmentation using multiple atlases may significantly improve segmentation accuracy.

“In recent years, methods that use the multi atlas label fusion framework have obtained state-of-the-art accuracy, showing that consensus segmentation using multiple atlases may significantly improve the segmentation accuracy, while making it also more robust to sporadic registration errors/imperfections (Iglesias and Sabuncu, 2015; Warfield, 2017).”

• cites Iglesias and Sabuncu, 2015 sourceSupported
• cites Warfield, 2017Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Various studies have reported high rates of failures resulting in exclusion of large number of scans, inconsistencies in segmentations and age related bias for Freesurfer.

“Although Freesurfer is extensively validated and it has been used in a large number of studies, various studies have also reported high rates of failures resulting in exclusion of large number of scans, inconsistencies in segmentations and age related bias (Wenger et al., 2014; Cherbuin et al., 2009).”

• cites Wenger et al., 2014 sourceNot checked — no full text of the source
• cites Cherbuin et al., 2009 sourcePartly supported

  Methods (Image analysis; page unavailable) reports that 27 scans could not be processed by the standard FreeSurfer pipeline, and Results (page unavailable) reports 13 hippocampal pairs excluded for major segmentation defects. The Discussion (page unavailable) describes differences between FreeSurfer and manual segmentations, but calls the number excluded for major defects small and says the study’s narrow age range limited its ability to detect an age-dependent effect. It therefore does not establish a high failure rate or age-related bias. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

High number of Freesurfer exclusions based on detailed visual verification of individual ROIs was previously reported.

“High number of Freesurfer exclusions based on detailed visual verification of individual ROIs was previously reported (Mccarthy 2015; Zandifar et al., 2017).”

• cites Mccarthy 2015 sourcePartly supported

  The Image analysis methods (page unavailable) describe detailed visual inspection of FreeSurfer-generated surfaces and manual correction of errors. The paper does not report a high number of scans or ROIs excluded as a result of that inspection; its mention of excluded studies in the Introduction concerns a literature review, not FreeSurfer quality control. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Zandifar et al., 2017 sourcePartly supported

  Experiment 2: results (p. 5) says FreeSurfer had the largest number of segmentation failures, and Limitations (p. 9) reports 101 subjects for whom the FreeSurfer pipeline could not estimate hippocampal volume. The paper says subjects were included only if they passed quality control, but it does not say the FreeSurfer exclusions resulted from detailed visual verification of individual ROIs. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S2 (checkable).

Inter-scanner reproducibility is an issue of rapidly rising significance with the emergence of large-scale meta/mega-analyses that pool data from multiple studies.

“The latter was a test of inter-scanner reproducibility, an issue that is of rapidly rising significance with the emergence of large-scale meta/mega-analyses that pool data from multiple studies (Thompson et al., 2014; Davatzikos, 2018).”

• cites Thompson et al., 2014 sourceSupported
• cites Davatzikos, 2018 sourceNot checked — no full text of the source

Classified as probably not a substantial claim.

Consensus segmentation using multiple atlases may make segmentation more robust to sporadic registration errors/imperfections.

“In recent years, methods that use the multi atlas label fusion framework have obtained state-of-the-art accuracy, showing that consensus segmentation using multiple atlases may significantly improve the segmentation accuracy, while making it also more robust to sporadic registration errors/imperfections (Iglesias and Sabuncu, 2015; Warfield, 2017).”

• cites Iglesias and Sabuncu, 2015 sourcePartly supported

  The Historical Introduction describes combining labels from multiple registered atlases by majority voting, which supports the idea of consensus segmentation. But §2.3 (Registration; page unavailable) does not establish greater robustness to sporadic registration errors: it cautions that poor candidates from suboptimal registrations can worsen fusion. The claimed robustness is plausible, but is not stated in the cited passages. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites Warfield, 2017Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Multi-atlas label fusion methods have obtained state-of-the-art accuracy in recent years.

“In recent years, methods that use the multi atlas label fusion framework have obtained state-of-the-art accuracy, showing that consensus segmentation using multiple atlases may significantly improve the segmentation accuracy, while making it also more robust to sporadic registration errors/imperfections (Iglesias and Sabuncu, 2015; Warfield, 2017).”

• cites Iglesias and Sabuncu, 2015 sourceSupported
• cites Warfield, 2017Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

The age bias in Freesurfer segmentations is more pronounced in Freesurfer-v5.3 than in Freesurfer-v6.0, but is still present in v6.0.

“This bias was more pronounced in Freesurfer-v5.3, which has been used in multiple previous studies of aging, while the effect was mitigated in more recent Freesurfer-v6.0, albeit still present.”

• Also stated in: § Analysis of hippocampal volume differences · plain

  “Freesurfer segmentation had a significantly higher slope with age (s_Freesurfer-v5.3 = −52.57,s_Freesurfer-v6.0 = −44.44).”

May combine two claims.

MUSE segmentations showed no bias with subject age or hippocampus volume.

“On the other hand, MUSE segmentations were more consistent with ground-truth segmentations, without bias with subject age or hippocampus volume.”

May combine two claims.

Hippocampus segmentations by Freesurfer-v5.3 showed a bias towards over-segmentation of larger hippocampi, and under-segmentation of hippocampi in smaller or older individuals.

“We found that hippocampus segmentations by both Freesurfer-v6.0 and Freesurfer-v5.3 showed a bias towards over-segmentation of larger hippocampi, and under-segmentation of hippocampi in smaller or older individuals, although this age bias was reduced with the newer version.”

May combine two claims.

In deep structures, MUSE obtained very consistent segmentations across subjects with higher correlations compared to both Freesurfer versions.

“While Freesurfer 6.0 was comparable to MUSE for cortical ROIs, in deep structures MUSE obtained very consistent segmentations across subjects with higher correlations compared to both Freesurfer versions.”

May combine two claims.

Hippocampus segmentations by Freesurfer-v6.0 showed a bias towards over-segmentation of larger hippocampi, and under-segmentation of hippocampi in smaller or older individuals.

“We found that hippocampus segmentations by both Freesurfer-v6.0 and Freesurfer-v5.3 showed a bias towards over-segmentation of larger hippocampi, and under-segmentation of hippocampi in smaller or older individuals, although this age bias was reduced with the newer version.”

May combine two claims.

For deep structures, the differences in inter-scanner correlations were significant between MUSE and Freesurfer-v5.3, but not for Freesurfer-v6.0.

“For the deep structures, the differences were significant between MUSE and Freesurfer-v5.3, but not for Freesurfer-v6.0 (Table 4).”

Classified as probably not a substantial claim.

P-values were corrected using FDR correction for multiple comparisons.

“P-values were corrected using FDR correction for multiple comparisons of same subject across different methods.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For Freesurfer segmentations, 2 cases from Dataset1 were flagged for exclusion due to gross errors.

“For the Freesurfer segmentations, 2 cases from Dataset1 were flagged for exclusion due to gross errors (suppl. Fig. 2).”

Classified as probably not a substantial claim. May combine two claims.

Matched ROIs were inspected by two manual raters and revisions were done after mutual agreement.

“Matched ROIs have been inspected by two manual raters (DS and MH) based on the ROI names and spatial correspondences between them. Revisions have been done after mutual agreement of the two raters.”

Probably not a substantial claim: may not pass S3 (load-bearing).

MUSE was the top-ranking method in the MICCAI-SATA challenge on deep brain segmentation.

“MUSE was the top-ranking method in the MICCAI-SATA challenge on deep brain segmentation (Asman et al., 2013)”

• cites Asman et al., 2013Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The final sample included 132 subjects for Dataset1 and 113 subjects for Dataset2 after exclusion of flagged cases.

“The final sample included 132 subjects for Dataset1 and 113 subjects for Dataset2 after exclusion of the flagged cases.”

Classified as probably not a substantial claim.

A binary QC flag (accept or exclude) was assigned to each segmentation based on the consensus of two independent raters.

“A binary QC flag, i.e. accept or exclude, was assigned to each segmentation, based on the consensus of two independent raters.”

May combine two claims.

The robustness of MUSE was expected, as multiple atlases provide various representations of the anatomy, while the label fusion of multiple warped atlases allows the method to correct the effect of individual registrations that failed, unless the failure is not systematic to a majority of the atlases.

“The robustness of MUSE was expected, as multiple atlases provide various representations of the anatomy, while the label fusion of multiple warped atlases allows the method to correct the effect of individual registrations that failed, unless the failure is not systematic to a majority of the atlases.”

Classified as probably not a substantial claim.

Because Freesurfer and MUSE use their own atlas sets with different ROI label denotations, a direct comparison of the two methods is not possible.

“Also, because Freesurfer and MUSE use their own atlas sets with different ROI label denotations, a direct comparison of the two methods is not possible.”

Probably not a substantial claim: may not pass S4 (based).

Even when the image registration fails for one or more atlas images, the voting between multiple atlases presumably helps obtain a correct segmentation, unless there is a systematic registration error that affects a majority of the atlases.

“The advantage of the multi atlas approach is twofold: a) multiple atlases allow capturing a broader anatomical variation, e.g. by including atlases from subjects with different sex and age, thereby allowing the label fusion algorithm to select subject-appropriate atlases on a regional basis; and b) even when the image registration fails for one or more atlas images, the voting between multiple atl”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Over 1600 neuroimaging publications used Freesurfer for quantification of brain volumes (as listed in pubmed.gov).

“Over 1600 neuroimaging publications used Freesurfer for quantification of brain volumes (as listed in pubmed.gov).”

May combine two claims.

The major version update in Freesurfer resulted in considerable differences in the final segmentation, improving the overall accuracy.

“suggesting that the major version update in Freesurfer resulted in considerable differences in the final segmentation, improving the overall accuracy.”

Probably not a substantial claim: may not pass S2 (checkable).

Multi-atlas segmentation has a great potential of becoming the standard approach for segmentation of brain images in population studies and in clinical applications.

“Given these favorable results and increasing availability of parallel and cloud computing capacities, multi-atlas segmentation has a great potential of becoming the standard approach for segmentation of brain images in population studies and in clinical applications.”

Classified as probably not a substantial claim.

We did not prefer to use these scans in our comparisons, as this would be biased towards MUSE, even with cross-validation.

“We did not prefer to use these scans in our comparisons, as this would be biased towards MUSE, even with cross-validation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Our experiments were limited to comparisons between Freesurfer and the multi atlas segmentation approach.

“Also, our experiments were limited to comparisons between Freesurfer and the multi atlas segmentation approach.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

One reason for Freesurfer's spurious age effects could be that Freesurfer does not use the population-based specific template.

“One reason could be that Freesurfer does not use the population-based specific template.”

Classified as probably not a substantial claim.

Multiple atlases allow capturing a broader anatomical variation, e.g. by including atlases from subjects with different sex and age, thereby allowing the label fusion algorithm to select subject-appropriate atlases on a regional basis.

“The advantage of the multi atlas approach is twofold: a) multiple atlases allow capturing a broader anatomical variation, e.g. by including atlases from subjects with different sex and age, thereby allowing the label fusion algorithm to select subject-appropriate atlases on a regional basis; and b) even when the image registration fails for one or more atlas images, the voting between multiple atl”

May combine two claims.

The advantage of using a model based (i.e. a-priori defined ground-truth ROI labels) consensus labeling in deep structures could explain MUSE's higher correlations, because tissue contrast alone is not informative enough to guide segmentation.

“This finding is consistent with previously reported state-of-the-art accuracy of MAS methods in segmentation of deep structures, and could be explained by the advantage of using a model based (i.e. a-priori defined ground-truth ROI labels) consensus labeling in these regions where the tissue contrast alone is not informative enough to guide the segmentation.”

Classified as probably not a substantial claim.

A major problem for a systematic comparison is the limited availability of ground truth segmentations.

“This work has also some limitations. A major problem for a systematic comparison is the limited availability of ground truth segmentations.”