CoChat

Toward naturalistic neuroscience: Mechanisms underlying the flattening of brain hierarchy in movie-watching compared to rest and task

Open access

The DOI matches its Crossref record; 105 of the 105 references it cites were found in a registry or an index.

Abstract

Identifying the functional specialization of the brain has moved from using cognitive tasks and resting state to using ecological relevant, naturalistic movies. We leveraged a large-scale neuroimaging dataset to directly investigate the hierarchical reorganization of functional brain activity when watching naturalistic films compared to performing seven cognitive tasks and resting. A thermodynamics-inspired whole-brain model paradigm revealed the generative underlying mechanisms for changing the balance in causal interactions between brain regions in different conditions. Paradoxically, the hierarchy is flatter for movie-watching, and the level of nonreversibility is significantly smaller in comparison to both rest and tasks, where the latter in turn have the highest levels of hierarchy and nonreversibility. The underlying mechanisms were revealed by the model-based generative effective connectivity (GEC). Naturalistic films could therefore provide a fast and convenient way to measure important changes in GEC (integrating functional and anatomical connectivity) found in, for example, neuropsychiatric disorders. Overall, this study demonstrates the benefits of moving toward a more naturalistic neuroscience.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1126/sciadv.ade6049

Cites

105 of 105 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1. GCAT framework for discovering underlying causal mechanisms of hierarchical organization. (A) Level of hierarchy is given by the level of asymmetry of causal interactions between brain regions arising from the breaking of the detailed balance. The upper subpanel shows a nonhierarchical system in full detailed balance and thus fully reversible over time, i.e., no change in production entropy, S. In contrast, the bottom subpanel shows a change in production entropy, reflecting the asymmetry of the underlying causal interactions. (B) Estimating the arrow of time requires the forward time series of each region (in black) and the time-reversed time series (in red). (C) Basic principle of how the level of NR can be computed through the pairwise level of asymmetry using a time-shifted measure of the correlations between the forward (x, y) (top row) and the reversed [x(r), y(r)] time series (bottom row). The difference between these time-shifted correlations provides a quantification of the asymmetry in the interactions between pairs of regions (for a given shift ∆t = T). (D) Hierarchy is computed through the generalization of the pairwise NR for the whole brain, i.e., as a matrix involving all pairs. The hierarchy is given by the NR matrix, which is the difference between the two time-shifted correlation matrices for the forward and reversed time series (at a given shift time point ∆t = T; see Methods). (E) NR matrix is used to fit a whole-brain model creating the GEC, which provides the underlying causal mechanisms (see Methods). (F) Average of the NR matrix provides a model-free estimate of the hierarchy that can be contrasted over conditions. (G) Further mechanistic insights into hierarchy can be provided by rendering the in and out degree of the GEC matrix.
Fig. 1. GCAT framework for discovering underlying causal mechanisms of hierarchical organization. (A) Level of hierarchy is given by the level of asymmetry of causal interactions between brain regions arising from the breaking of the detailed balance. The upper subpanel shows a nonhierarchical system in full detailed balance and thus fully reversible over time, i.e., no change in production entropy, S. In contrast, the bottom subpanel shows a change in production entropy, reflecting the asymmetry of the underlying causal interactions. (B) Estimating the arrow of time requires the forward time series of each region (in black) and the time-reversed time series (in red). (C) Basic principle of how the level of NR can be computed through the pairwise level of asymmetry using a time-shifted measure of the correlations between the forward (x, y) (top row) and the reversed [x(r), y(r)] time series (bottom row). The difference between these time-shifted correlations provides a quantification of the asymmetry in the interactions between pairs of regions (for a given shift ∆t = T). (D) Hierarchy is computed through the generalization of the pairwise NR for the whole brain, i.e., as a matrix involving all pairs. The hierarchy is given by the NR matrix, which is the difference between the two time-shifted correlation matrices for the forward and reversed time series (at a given shift time point ∆t = T; see Methods). (E) NR matrix is used to fit a whole-brain model creating the GEC, which provides the underlying causal mechanisms (see Methods). (F) Average of the NR matrix provides a model-free estimate of the hierarchy that can be contrasted over conditions. (G) Further mechanistic insights into hierarchy can be provided by rendering the in and out degree of the GEC matrix.CC BY-NC · PubMed Central
Fig. 2. Different functional hierarchies for movies, rest, and tasks. We estimated the functional hierarchy as characterized by the levels of NR. (A) Direct comparison of hierarchy in naturalistic movie, rest, and tasks shows that movie-watching (averaged over all sessions) has a significantly more flattened hierarchy (i.e., lower NR) compared to both rest and tasks (average over all seven tasks). Rest is less nonreversible than movie-watching but significantly less than task. (B) Each session of movie-watching (measured with 7-T fMRI) is significantly less nonreversible than rest. (C) Equally, all seven cognitive tasks (measured with 3-T fMRI) are significantly more hierarchically (i.e., nonreversible) than rest, suggesting the importance of hierarchy for computation.
Fig. 2. Different functional hierarchies for movies, rest, and tasks. We estimated the functional hierarchy as characterized by the levels of NR. (A) Direct comparison of hierarchy in naturalistic movie, rest, and tasks shows that movie-watching (averaged over all sessions) has a significantly more flattened hierarchy (i.e., lower NR) compared to both rest and tasks (average over all seven tasks). Rest is less nonreversible than movie-watching but significantly less than task. (B) Each session of movie-watching (measured with 7-T fMRI) is significantly less nonreversible than rest. (C) Equally, all seven cognitive tasks (measured with 3-T fMRI) are significantly more hierarchically (i.e., nonreversible) than rest, suggesting the importance of hierarchy for computation.CC BY-NC · PubMed Central
Fig. 3. Whole-brain model provides causal insights into the functional hierarchy of movie-watching. The figure shows how to identify the underlying causal mechanisms resulting in the different levels of hierarchy (i.e., NR) when movie-watching. (A) Procedure starts with fitting a whole-brain model, initially using the anatomical connectivity and then iteratively adjusting a GEC according to either fitting this to the empirical FC alone (GFC) or including the NR (GNR) matrices. (B) Upper row shows the optimization of the whole-brain model based on optimizing only with FC, while the lower row shows the same but including the optimization with NR. As can be seen in the leftmost panels, the evolution of learning improves the level of fit to FC (correlation between empirical and simulated matrices, black curves) in both cases but only fitting the FC does not give a good fit to the empirical NR (see red curves). The second column of panels show the optimized GEC matrices (GFC and GNR). While difficult to discern, the former is symmetrical, while the latter is asymmetrical, as quantified below. The third column of panels shows the simulated NR matrices, and the level of fit to the empirical NR is shown in the scatterplot in the fourth column of panels. It is very clear that only the GNR optimization is able to capture the level of empirical NR and consequently, the hierarchy. (C) Leftmost figure is quantifying the level of asymmetry of the GNR. As can be seen in the boxplot, there is no asymmetry for GFC, but strong asymmetry for GNR. This is further explored in the inset, which shows that there are many asymmetric pairs of brain regions. Last, we visualize these asymmetries with the full and thresholded matrices.
Fig. 3. Whole-brain model provides causal insights into the functional hierarchy of movie-watching. The figure shows how to identify the underlying causal mechanisms resulting in the different levels of hierarchy (i.e., NR) when movie-watching. (A) Procedure starts with fitting a whole-brain model, initially using the anatomical connectivity and then iteratively adjusting a GEC according to either fitting this to the empirical FC alone (GFC) or including the NR (GNR) matrices. (B) Upper row shows the optimization of the whole-brain model based on optimizing only with FC, while the lower row shows the same but including the optimization with NR. As can be seen in the leftmost panels, the evolution of learning improves the level of fit to FC (correlation between empirical and simulated matrices, black curves) in both cases but only fitting the FC does not give a good fit to the empirical NR (see red curves). The second column of panels show the optimized GEC matrices (GFC and GNR). While difficult to discern, the former is symmetrical, while the latter is asymmetrical, as quantified below. The third column of panels shows the simulated NR matrices, and the level of fit to the empirical NR is shown in the scatterplot in the fourth column of panels. It is very clear that only the GNR optimization is able to capture the level of empirical NR and consequently, the hierarchy. (C) Leftmost figure is quantifying the level of asymmetry of the GNR. As can be seen in the boxplot, there is no asymmetry for GFC, but strong asymmetry for GNR. This is further explored in the inset, which shows that there are many asymmetric pairs of brain regions. Last, we visualize these asymmetries with the full and thresholded matrices.CC BY-NC · PubMed Central
Fig. 4. Identifying the underlying causal drivers of hierarchy changes in movies, rest and tasks. We can determine the information flow in terms of drivers and receivers by using the matrices of the GEC, obtained through using a whole-brain model for movies, rest, and tasks (fitted to both the model-free measures of NR and FC). This provides direct measures of the underlying brain hierarchy. (A) For all conditions, the figures show the receivers (incoming, Gin), drivers (outgoing, Gout), and their sum (Gtotal). The hierarchy is given by the gradations in color and, as can be seen movie, has a significantly lower hierarchy (deeper red) than both rest (orange) and task (strongest yellow). (B) Decrease in hierarchy in movies versus rest can be explicit shown by rendering the difference between their respective Gtotal. As can be seen from the colormap, where negative values are more blue and positive values are more yellow, only prefrontal and some visual regions are stronger for movies than rest. In other words, while the general hierarchy is flattened in movies, the prefrontal and visual regions are more nonreversible for movies, suggesting that the computation performed by these regions drives the breaking of the detailed balance when movie-watching. In contrast, when comparing the Gtotal for the average over all tasks versus rest, the general hierarchy is significantly larger for tasks and the main drivers of the computation is again found primarily in the prefrontal regions. (C) Confirming the role of prefrontal cortex in driving the breaking of the detailed balance, we computed the intersection of the top 50% regions of the contrasts shown in (B). This showed that primarily prefrontal regions (as well as some parietal, visual, and temporal) are the common drivers orchestrating computation in the brain.
Fig. 4. Identifying the underlying causal drivers of hierarchy changes in movies, rest and tasks. We can determine the information flow in terms of drivers and receivers by using the matrices of the GEC, obtained through using a whole-brain model for movies, rest, and tasks (fitted to both the model-free measures of NR and FC). This provides direct measures of the underlying brain hierarchy. (A) For all conditions, the figures show the receivers (incoming, Gin), drivers (outgoing, Gout), and their sum (Gtotal). The hierarchy is given by the gradations in color and, as can be seen movie, has a significantly lower hierarchy (deeper red) than both rest (orange) and task (strongest yellow). (B) Decrease in hierarchy in movies versus rest can be explicit shown by rendering the difference between their respective Gtotal. As can be seen from the colormap, where negative values are more blue and positive values are more yellow, only prefrontal and some visual regions are stronger for movies than rest. In other words, while the general hierarchy is flattened in movies, the prefrontal and visual regions are more nonreversible for movies, suggesting that the computation performed by these regions drives the breaking of the detailed balance when movie-watching. In contrast, when comparing the Gtotal for the average over all tasks versus rest, the general hierarchy is significantly larger for tasks and the main drivers of the computation is again found primarily in the prefrontal regions. (C) Confirming the role of prefrontal cortex in driving the breaking of the detailed balance, we computed the intersection of the top 50% regions of the contrasts shown in (B). This showed that primarily prefrontal regions (as well as some parietal, visual, and temporal) are the common drivers orchestrating computation in the brain.CC BY-NC · PubMed Central
Fig. 5. GEC is excellent for classification of movie-watching. We used machine learning to classify movie-watching compared to rest and between different movies using either GEC or FC. (A) Figure shows the boxplots of the classification performance (across 100-fold) and the associated average confusion matrix. As can be seen, the classification is much better when using GEC than using FC (with the performance numbers shown on the figure). (B) Similarly, the classification of specific movie extracts (Hollywood or CC license on Vimeo) is also much better when using GEC than using FC. Here, using classification with GEC is significantly better than using FC. As can be seen, the average performance for GEC is 93.4%, while the average performance using FC is closer to chance levels (62.7%). Overall, the GEC provides the causal mechanistic principles for a given condition and therefore is excellent for classification of that condition.
Fig. 5. GEC is excellent for classification of movie-watching. We used machine learning to classify movie-watching compared to rest and between different movies using either GEC or FC. (A) Figure shows the boxplots of the classification performance (across 100-fold) and the associated average confusion matrix. As can be seen, the classification is much better when using GEC than using FC (with the performance numbers shown on the figure). (B) Similarly, the classification of specific movie extracts (Hollywood or CC license on Vimeo) is also much better when using GEC than using FC. Here, using classification with GEC is significantly better than using FC. As can be seen, the average performance for GEC is 93.4%, while the average performance using FC is closer to chance levels (62.7%). Overall, the GEC provides the causal mechanistic principles for a given condition and therefore is excellent for classification of that condition.CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 105 references

Research works printed but not linked: 3

  1. 10.1002/(SICI)1097-0193(1997)5:4<317::AID-HBM19>3.0.CO;2-A · DOI
  2. B. J. Baars A Cognitive Theory of Consciousness (Cambridge Univ. Press 1989).
  3. Y. A. Kuznetsov Elements of applied bifurcation theory (Springer 1998).

Cited by

See all 105

Metadata

Authors
Kringelbach, M. L., Perl, Y. S., Tagliazucchi, E., & Deco, G.
Affiliations
  • Aarhus University (DK)
  • Universitat Pompeu Fabra (ES)
  • Adolfo Ibáñez University (CL)
  • Institució Catalana de Recerca i Estudis Avançats (ES)
Corresponding author
  • Kringelbach, Morten L.
  • Deco, Gustavo
Venue
Science Advances
Published
2023-01-13
Volume
9
Issue
2
Pages
eade6049-eade6049
ISSN
2375-2548 (ISSN-L)
OpenAlex
W4316016059
Cited by
105
References
104
Citations by year
2026: 21 · 2025: 37 · 2024: 32 · 2023: 15
FWCI
15.72
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • Science Advances (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • Repositori digital de la UPF (Universitat Pompeu Fabra) (submittedVersion, cc-by-nc-nd)
Funders
  • Ministerio de Ciencia, Innovación y Universidades (10.13039.501100011033, / AEI10.13039/501100011033, PID2019, 13039, PID2019-105772GB-I00/AEI/10.13039/501100011033, 10.13039, .13039/501100011033, 501100011033)
  • European Commission (10.13039/501100011033, 13039/501100011033, 501100011033, HORIZON2020, /AEI/10.13039/501100011033, 896354, 945539)
  • National Research Foundation (DNRF117)
  • Deutsche Forschungsgemeinschaft (945539)
  • Danmarks Grundforskningsfond (DNRF117)
  • Agencia Nacional de Promoción de la Investigación, el Desarrollo Tecnológico y la Innovación (PICT-2018, PICT-2019-02294)
  • Agencia Estatal de Investigación (AEI//10.13039/501100011033/, 501100011033, 10.13039/501100011033, 13039, .13039/501100011033, 10.13039, 945539, AEI/10, I00/AEI/10.13039/501100011033, PID2019)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neural dynamics and brain function
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced Neuroimaging Techniques and Applications
Keywords
brain hierarchy, effective connectivity, functional specialization, resting state, cognitive tasks, functional connectivity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.