CoChat

Mapping between fMRI responses to movies and their natural language annotations

Open access
Vodrahalli, Kiran, Chen, Po-Hsuan, Liang, Yingyu, Baldassano, Christopher, Chen, Janice, Yong, Esther, Honey, Christopher, Hasson, Uri, Ramadge, Peter, Norman, Kenneth A., Arora, Sanjeev · 2018 · NeuroImage
Cited by 75

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.

Abstract

Several research groups have shown how to map fMRI responses to the meanings of presented stimuli. This paper presents new methods for doing so when only a natural language annotation is available as the description of the stimulus. We study fMRI data gathered from subjects watching an episode of BBCs Sherlock (Chen et al., 2017), and learn bidirectional mappings between fMRI responses and natural language representations. By leveraging data from multiple subjects watching the same movie, we were able to perform scene classification with 72% accuracy (random guessing would give 4%) and scene ranking with average rank in the top 4% (random guessing would give 50%). The key ingredients underlying this high level of performance are (a) the use of the Shared Response Model (SRM) and its variant SRM-ICA (Chen et al., 2015; Zhang et al., 2016) to aggregate fMRI data from multiple subjects, both of which are shown to be superior to standard PCA in producing low-dimensional representations for the tasks in this paper; (b) a sentence embedding technique adapted from the natural language processing (NLP) literature (Arora et al., 2017) that produces semantic vector representation of the annotations; (c) using previous timestep information in the featurization of the predictor data. These optimizations in how we featurize the fMRI data and text annotations provide a substantial improvement in classification performance, relative to standard approaches.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.neuroimage.2017.06.042

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 11 references

Research works printed but not linked: 14

  1. Anderson, M.J., Capota, M., Turek, J.S., Zhu, X., Willke, T.L., Wang, Y., Chen, P.-H., Manning, J.R., Ramadge, P.J., Norman, K.A., 2016. Enabling factor analysis on thousand-subject neuroimaging datasets URL http://arXiv:1608.04647arXiv:1608.04647. · DOI
  2. Arora, S., Liang, Y., Ma, T. 2017. A Simple but Tough-to-Beat Baseline for Sentence Embeddings, International Conference on Learning Representations (ICLR) URL 〈https://openreview.net/pdf?Id=SyK00v5xx〉.
  3. Chen, P.-H., Chen, J., Yeshurun, Y., Hasson, U., Haxby, J.V., Ramadge, P.J., 2015. A Reduced-Dimension fMRI Shared Response Model, Proceedings of the 29th Annual Conference on Neural Information Processing Systems (NIPS).
  4. Conroy, 2013, Inter-subject alignment of human cortical anatomy using functional connectivity, NeuroImage, 81, 400, 10.1016/j.neuroimage.2013.05.009 · DOI · Paper page
  5. Deerwester, 1990, Indexing by latent semantic analysis, J. Am. Soc. Inf. Sci., 41, 391, 10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9 · DOI · Paper page
  6. Hasson, 2010, Reliability of cortical activity during natural stimulation, Trends Cogn. Sci., 14, 40, 10.1016/j.tics.2009.10.011 · DOI · Paper page
  7. Haxby, 2011, A common, high-dimensional model of the representation space in human ventral temporal cortex, Neuron, 72, 404, 10.1016/j.neuron.2011.08.026 · DOI · Paper page
  8. Huth, 2012, A continuous semantic space describes the representation of thousands of object and action categories across the human brain, Neuron, 76, 1210, 10.1016/j.neuron.2012.10.014 · DOI · Paper page
  9. Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R.S., Torralba, A., Urtasun, R., Fidler, S., 2015. Skip-Thought Vectors, Advances in Neural Information Processing Systems URL 〈https://papers.nips.cc/paper/5950-skip-thought-vectors〉.
  10. Le, Q., Mikolov, T., 2014. Distributed Representations of Sentences and Documents, Proceedings of the 31st International Conference on Machine Learning, JMLR 32.
  11. Pereira, 2011, Generating text from functional brain images, Front. Human. Neurosci., 5, 72, 10.3389/fnhum.2011.00072 · DOI · Paper page
  12. Pereira, F., Lou, B., Pritchett, B., Kanwisher, N., Botvinick, M., Fedorenko, E., 2016. Decoding of generic mental representations from functional MRI data using word embeddings, bioRxiv preprint. · DOI
  13. Wehbe, L., Murphy, B., Talukdar, P., Fyshe, A., Ramdas, A., Mitchell, T., 2014. Simultaneously Uncovering the Patterns of Brain Regions Involved in Different Story Reading Subprocesses, PLOS ONE 9. · DOI · Paper page
  14. Zhang, H., Chen, P.-H., Chen, J., Zhu, X., Turek, J.S., Willke, T.L., Hasson, U., Ramadge, P.J., 2016. A searchlight factor model approach for locating shared information in multi-subject fmri analysis, arXiv preprint URL http://arXiv:1609.09432arXiv:1609.09432.

Cited by

See all 75

Metadata

Authors
Vodrahalli, K., Chen, P.-H., Liang, Y., Baldassano, C., Chen, J., Yong, E., Honey, C., Hasson, U., Ramadge, P., Norman, K. A., & Arora, S.
Venue
NeuroImage
Publisher
Elsevier BV
Published
2018-10
Volume
180
Issue
Pt A
Pages
223-231
ISSN
1053-8119 (ISSN-L)
OpenAlex
W2598802110
Cited by
75
References
11
FWCI
4.01
Type
journal article
Language
en
Open access
hybrid
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Mapping between fMRI responses to movies and their natural language annotations. https://app.cochat.ai/papers/10.1016/j.neuroimage.2017.06.042
The paper
Vodrahalli, K., Chen, P.-H., Liang, Y., Baldassano, C., Chen, J., Yong, E., Honey, C., Hasson, U., Ramadge, P., Norman, K. A., & Arora, S. (2018). Mapping between fMRI responses to movies and their natural language annotations. NeuroImage. https://doi.org/10.1016/j.neuroimage.2017.06.042

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.