The DOI matches its Crossref record; the references it cites are not checked yet.
Large language models predict human sensory judgments across six modalities
Open accessAbstract
Determining the extent to which the perceptual world can be recovered from language is a longstanding problem in philosophy and cognitive science. We show that state-of-the-art large language models can unlock new insights into this problem by providing a lower bound on the amount of perceptual information that can be extracted from language. Specifically, we elicit pairwise similarity judgments from GPT models across six psychophysical datasets. We show that the judgments are significantly correlated with human data across all domains, recovering well-known representations like the color wheel and pitch spiral. Surprisingly, we find that a model (GPT-4) co-trained on vision and language does not necessarily lead to improvements specific to the visual modality, and provides highly correlated predictions with human data irrespective of whether direct visual input is provided or purely textual descriptors. To study the impact of specific languages, we also apply the models to a multilingual color-naming task. We find that GPT-4 replicates cross-linguistic variation in English and Russian illuminating the interaction of language and perception.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In contrast, the task-based cognitive approaches discussed earlier in this paper36,38,40–43 do not commit this error. They use established experimental paradigms to empirically test whether human and LLM cognition converge or diverge on specific dimensions, without presupposing cognitive equivalence as a starting point.”
From Understanding large language models demands distinguishing human projection from machine cognition (2026) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
The citation lists are queued
They are fetched later and appear here then.
References 44 · Cited by 73
Research works printed but not linked: 7
- Forster, M. & von Herder, J.G. In The Stanford Encyclopedia of Philosophy (Zalta, E. N. & Nodelman, U. eds.). Winter 2023 Ed. (Metaphysics Research Lab, Stanford University, 2023).
- Patel, R. & Pavlick, E. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations (2021).
- Marjieh, R. et al. Words are all you need? capturing human sensory similarity with textual descriptors. In The Eleventh International Conference on Learning Representations (2022).
- Berlin, B. & Kay, P. Basic Color Terms: Their Universality and Evolution (University of California Press, 1991).
- Tucker, M., Levy, R., Shah, J. A. & Zaslavsky, N. Trading off utility, informativeness, and complexity in emergent communication. Adv. Neural Inf. Process. Syst. 35, 22214–22228 (2022).
- Harrison, P. et al. Gibbs sampling with people. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F. & Lin, H. (eds.) Advances in Neural Information Processing Systems. Vol. 33. 10659–10671 (Curran Associates, Inc., 2020).
- Brown, W. Some experimental results in the correlation of mental abilities 1. Br. J. Psychol. 1904–1920(3), 296–322 (1910).
Metadata
- Authors
- Marjieh, R., Sucholutsky, I., van Rijn, P., Jacoby, N., & Griffiths, T. L.
- Affiliations
- Princeton University (US)
- Cornell University (US)
- Corresponding author
- Marjieh, Raja
- Venue
- Scientific Reports
- Published
- 2024-09-13
- Volume
- 14
- Issue
- 1
- Pages
- 21445-21445
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC11399123
- OpenAlex
- W4402512774
- Cited by
- 73
- References
- 44
- Citations by year
- 2026: 24 · 2025: 31 · 2024: 12
- FWCI
- 27.17
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Scientific Reports (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- This research was supported by the NOMIS Foundation, and an NSERC fellowship to author IS (567554-2022)
- Topics
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Keywords
- large language models, cross-modal perception, color wheel, language and perception, cross-linguistic variation, color naming, GPT-4, vision-language models, psychophysical data
- MeSH terms
- Humans, Judgment, Language, Visual Perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Large language models predict human sensory judgments across six modalities. https://app.cochat.ai/papers/10.1038/s41598-024-72071-1
- The paper
- Marjieh, R., Sucholutsky, I., van Rijn, P., Jacoby, N., & Griffiths, T. L. (2024). Large language models predict human sensory judgments across six modalities. Scientific Reports. https://doi.org/10.1038/s41598-024-72071-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.