Mapping between fMRI responses to movies and their natural language annotations
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Several research groups have shown how to map fMRI responses to the meanings of presented stimuli. This paper presents new methods for doing so when only a natural language annotation is available as the description of the stimulus. We study fMRI data gathered from subjects watching an episode of BBCs Sherlock (Chen et al., 2017), and learn bidirectional mappings between fMRI responses and natural language representations. By leveraging data from multiple subjects watching the same movie, we were able to perform scene classification with 72% accuracy (random guessing would give 4%) and scene ranking with average rank in the top 4% (random guessing would give 50%). The key ingredients underlying this high level of performance are (a) the use of the Shared Response Model (SRM) and its variant SRM-ICA (Chen et al., 2015; Zhang et al., 2016) to aggregate fMRI data from multiple subjects, both of which are shown to be superior to standard PCA in producing low-dimensional representations for the tasks in this paper; (b) a sentence embedding technique adapted from the natural language processing (NLP) literature (Arora et al., 2017) that produces semantic vector representation of the annotations; (c) using previous timestep information in the featurization of the predictor data. These optimizations in how we featurize the fMRI data and text annotations provide a substantial improvement in classification performance, relative to standard approaches.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Intersubject synchronization of cortical activity during natural vision
Hasson, U., Nir, Y., Levy, I., Fuhrmann, G., & Malach, R. (2004) · Science · doi:10.1126/science.1089506 · Cited by 1,839 · cited 3×
“In order to de fi ne the DMN-A and DMN-B regions, as well as the Ventral and Dorsal language area regions, the intersubject functional correlation matrix (Hasson et al., 2004) was calculated from the fMRI data of 36 subjects collected while they were listening to stories (Simony et al., 2016).”
BackgroundMethodSelf-citationNatural speech reveals the semantic maps that tile human cerebral cortex
Huth, A. G., de Heer, W. A., Griffiths, T. L., Theunissen, F. E., & Gallant, J. L. (2016) · Nature · doi:10.1038/nature17637 · Cited by 1,504
Predicting Human Brain Activity Associated with the Meanings of Nouns
Mitchell, T. M., Shinkareva, S. V., Carlson, A., Chang, K.-M., Malave, V. L., Mason, R. A., & Just, M. A. (2008) · Science · doi:10.1126/science.1152876 · Cited by 1,180
Discovering Event Structure in Continuous Narrative Perception and Memory
Baldassano, C., Chen, J., Zadbood, A., Pillow, J. W., Hasson, U., & Norman, K. A. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.041 · Cited by 952
Self-citationShared memories reveal shared structure in neural activity across individuals
Chen, J., Leong, Y. C., Honey, C. J., Yong, C. H., Norman, K. A., & Hasson, U. (2017) · Nature Neuroscience · doi:10.1038/nn.4450 · Cited by 645
Self-citationDynamic reconfiguration of the default mode network during narrative comprehension
Simony, E., Honey, C. J., Chen, J., Lositsky, O., Yeshurun, Y., Wiesel, A., & Hasson, U. (2016) · Nature Communications · doi:10.1038/ncomms12141 · Cited by 588 · cited 7×
“…DMN-A and DMN-B networks were identi fi ed by comparing the resultant clusters to the DMN ROIs derived by thresholding the functional correlation between the posterior cingulate (identi fi ed from an atlas) and the rest of the brain in resting-state fMRI data from 36 subjects (Simony et al., 2016).”
BackgroundMethodResultInfluentialSelf-citationSame Story, Different Story: The Neural Representation of Interpretive Frameworks
Yeshurun, Y., Swanson, S., Simony, E., Chen, J., Lazaridi, C., Honey, C. J., & Hasson, U. (2017) · Psychological Science · doi:10.1177/0956797616682029 · Cited by 300 · cited 2×
“We are interested in the DMN ROIs in particular since prior work has demonstrated that these regions play a crucial role in tracking the narrative in settings such as watching movies or reading stories (Hasson et al., 2004, 2010; Honey et al., 2012; Regev et al., 2013; Ames et al., 2015; Simony et al., 2016; Yeshurun…”
Self-citationA Latent Variable Model Approach to PMI-based Word Embeddings
Arora, S., Li, Y., Liang, Y., Ma, T., & Risteski, A. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00106 · Cited by 286 · cited 6×
“We train word embeddings as described in Arora et al. (2016), which perform on par with other standard word embedding techniques like GloVe and Word2Vec (Arora et al., 2016).”
BackgroundMethodInfluentialSelf-citationSelective and invariant neural responses to spoken and written narratives
Regev, M., Honey, C. J., Simony, E., & Hasson, U. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.1580-13.2013 · Cited by 215 · cited 3×
“In particular, the Whole Brain and the DMN regions perform best, supporting previous work by Regev et al. (2013) and others demonstrating that the DMN plays an important role in narrative processing.”
BackgroundMethodSelf-citationNot lost in translation: neural responses shared across languages
Honey, C. J., Thompson, C. R., Lerner, Y., & Hasson, U. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.1800-12.2012 · Cited by 192
Self-citation
Research works printed but not linked: 14
- Anderson, M.J., Capota, M., Turek, J.S., Zhu, X., Willke, T.L., Wang, Y., Chen, P.-H., Manning, J.R., Ramadge, P.J., Norman, K.A., 2016. Enabling factor analysis on thousand-subject neuroimaging datasets URL http://arXiv:1608.04647arXiv:1608.04647. · DOI
- Arora, S., Liang, Y., Ma, T. 2017. A Simple but Tough-to-Beat Baseline for Sentence Embeddings, International Conference on Learning Representations (ICLR) URL 〈https://openreview.net/pdf?Id=SyK00v5xx〉.
- Chen, P.-H., Chen, J., Yeshurun, Y., Hasson, U., Haxby, J.V., Ramadge, P.J., 2015. A Reduced-Dimension fMRI Shared Response Model, Proceedings of the 29th Annual Conference on Neural Information Processing Systems (NIPS).
- Conroy, 2013, Inter-subject alignment of human cortical anatomy using functional connectivity, NeuroImage, 81, 400, 10.1016/j.neuroimage.2013.05.009 · DOI · Paper page
- Deerwester, 1990, Indexing by latent semantic analysis, J. Am. Soc. Inf. Sci., 41, 391, 10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9 · DOI · Paper page
- Hasson, 2010, Reliability of cortical activity during natural stimulation, Trends Cogn. Sci., 14, 40, 10.1016/j.tics.2009.10.011 · DOI · Paper page
- Haxby, 2011, A common, high-dimensional model of the representation space in human ventral temporal cortex, Neuron, 72, 404, 10.1016/j.neuron.2011.08.026 · DOI · Paper page
- Huth, 2012, A continuous semantic space describes the representation of thousands of object and action categories across the human brain, Neuron, 76, 1210, 10.1016/j.neuron.2012.10.014 · DOI · Paper page
- Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R.S., Torralba, A., Urtasun, R., Fidler, S., 2015. Skip-Thought Vectors, Advances in Neural Information Processing Systems URL 〈https://papers.nips.cc/paper/5950-skip-thought-vectors〉.
- Le, Q., Mikolov, T., 2014. Distributed Representations of Sentences and Documents, Proceedings of the 31st International Conference on Machine Learning, JMLR 32.
- Pereira, 2011, Generating text from functional brain images, Front. Human. Neurosci., 5, 72, 10.3389/fnhum.2011.00072 · DOI · Paper page
- Pereira, F., Lou, B., Pritchett, B., Kanwisher, N., Botvinick, M., Fedorenko, E., 2016. Decoding of generic mental representations from functional MRI data using word embeddings, bioRxiv preprint. · DOI
- Wehbe, L., Murphy, B., Talukdar, P., Fyshe, A., Ramdas, A., Mitchell, T., 2014. Simultaneously Uncovering the Patterns of Brain Regions Involved in Different Story Reading Subprocesses, PLOS ONE 9. · DOI · Paper page
- Zhang, H., Chen, P.-H., Chen, J., Zhu, X., Turek, J.S., Willke, T.L., Hasson, U., Ramadge, P.J., 2016. A searchlight factor model approach for locating shared information in multi-subject fmri analysis, arXiv preprint URL http://arXiv:1609.09432arXiv:1609.09432.
Cited by
Discovering Event Structure in Continuous Narrative Perception and Memory
Baldassano, C., Chen, J., Zadbood, A., Pillow, J. W., Hasson, U., & Norman, K. A. (2017) · Neuron · doi:10.1016/j.neuron.2017.06.041 · Cited by 952
Self-citationRepresentation of Real-World Event Schemas during Narrative Perception
Baldassano, C., Hasson, U., & Norman, K. A. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.0251-18.2018 · Cited by 330
Self-citationTeacher-student neural coupling during teaching and learning
Nguyen, M., Chang, A., Micciche, E., Meshulam, M., Nastase, S. A., & Hasson, U. (2020) · doi:10.1101/2020.05.07.082958 · Cited by 67
Self-citationLeveraging shared connectivity to aggregate heterogeneous datasets into a common response space
Nastase, S. A., Liu, Y.-F., Hillman, H., Norman, K. A., & Hasson, U. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.116865 · Cited by 42
Self-citationThe promise of awake behaving infant fMRI as a deep measure of cognition
Yates, T. S., Ellis, C. T., & Turk-Browne, N. B. (2021) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2020.11.007 · Cited by 29
“Indeed, from task-free movie-watching alone, researchers can construct a semantic space [48] and provide a text caption of the current scene [49].”
Background
Metadata
- Authors
- Vodrahalli, K., Chen, P.-H., Liang, Y., Baldassano, C., Chen, J., Yong, E., Honey, C., Hasson, U., Ramadge, P., Norman, K. A., & Arora, S.
- Venue
- NeuroImage
- Publisher
- Elsevier BV
- Published
- 2018-10
- Volume
- 180
- Issue
- Pt A
- Pages
- 223-231
- ISSN
- 1053-8119 (ISSN-L)
- OpenAlex
- W2598802110
- Cited by
- 75
- References
- 11
- FWCI
- 4.01
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0/ (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Mapping between fMRI responses to movies and their natural language annotations. https://app.cochat.ai/papers/10.1016/j.neuroimage.2017.06.042
- The paper
- Vodrahalli, K., Chen, P.-H., Liang, Y., Baldassano, C., Chen, J., Yong, E., Honey, C., Hasson, U., Ramadge, P., Norman, K. A., & Arora, S. (2018). Mapping between fMRI responses to movies and their natural language annotations. NeuroImage. https://doi.org/10.1016/j.neuroimage.2017.06.042
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.