Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; the references it cites are not checked yet.

Large language models predict human sensory judgments across six modalities

Open access
Marjieh, Raja, Sucholutsky, Ilia, van Rijn, Pol, Jacoby, Nori, Griffiths, Thomas L. · 2024 · Scientific Reports

Abstract

Determining the extent to which the perceptual world can be recovered from language is a longstanding problem in philosophy and cognitive science. We show that state-of-the-art large language models can unlock new insights into this problem by providing a lower bound on the amount of perceptual information that can be extracted from language. Specifically, we elicit pairwise similarity judgments from GPT models across six psychophysical datasets. We show that the judgments are significantly correlated with human data across all domains, recovering well-known representations like the color wheel and pitch spiral. Surprisingly, we find that a model (GPT-4) co-trained on vision and language does not necessarily lead to improvements specific to the visual modality, and provides highly correlated predictions with human data irrespective of whether direct visual input is provided or purely textual descriptors. To study the impact of specific languages, we also apply the models to a multilingual color-naming task. We find that GPT-4 replicates cross-linguistic variation in English and Russian illuminating the interaction of language and perception.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-024-72071-1

Cites

Statistics

Figures

Fig. 1 (a) Schematic of the LLM-based and human similarity judgment elicitation paradigms. (b) Correlations between models and human data across six perceptual modalities, namely, pitch, loudness, colors, consonants, taste, and timbre (Pearson r; 95% CIs).
Fig. 1 (a) Schematic of the LLM-based and human similarity judgment elicitation paradigms. (b) Correlations between models and human data across six perceptual modalities, namely, pitch, loudness, colors, consonants, taste, and timbre (Pearson r; 95% CIs).CC BY · PubMed Central
Fig. 2 (a) Human and LLM similarity marginals and an example GPT-3 corresponding similarity matrix and its three-dimensional MDS solution for pitch. (b) MDS solutions for vocal consonants and colors for GPT-4 similarity matrices. To illustrate the structure of the results, we highlighted consonants with the same place of articulation in the vocal tract with the same shape and color, and added a rotated HSV color wheel for the color MDS.
Fig. 2 (a) Human and LLM similarity marginals and an example GPT-3 corresponding similarity matrix and its three-dimensional MDS solution for pitch. (b) MDS solutions for vocal consonants and colors for GPT-4 similarity matrices. To illustrate the structure of the results, we highlighted consonants with the same place of articulation in the vocal tract with the same shape and color, and added a rotated HSV color wheel for the color MDS.CC BY · PubMed Central
Fig. 3 (a) Example GPT-4 explanations for similarity judgment scores. (b) Word clouds for GPT-4 explanations in the domain of pitch, vocal consonants, and colors.
Fig. 3 (a) Example GPT-4 explanations for similarity judgment scores. (b) Word clouds for GPT-4 explanations in the domain of pitch, vocal consonants, and colors.CC BY · PubMed Central
Fig. 4 Color naming experiment using 330 Munsell colors from the World Color Survey (top, color space). (a) Adjusted Rand index illustrating the alignment between human and LLM experiments (95% CIs). The dashed lines for English represent lab-based free naming and forced-choice naming experiments collected by Lindsey and Brown35 (data reproduced with permission). (b) Data comparison between humans and LLMs in Russian and English. Participants and LLMs were shown colors and were asked to choose from the same 15-color list. The count of chosen colors for each option is given in parentheses. The color of a response cluster in the maps represents its average color (see Supplementary Fig. S3 for all maps). Colors for which less than 50% and 90% of the times the dominant color term was selected were indicated by “−” and “*”, respectively. If the dominant color term was selected more than 90% of the time, no marking was used.
Fig. 4 Color naming experiment using 330 Munsell colors from the World Color Survey (top, color space). (a) Adjusted Rand index illustrating the alignment between human and LLM experiments (95% CIs). The dashed lines for English represent lab-based free naming and forced-choice naming experiments collected by Lindsey and Brown35 (data reproduced with permission). (b) Data comparison between humans and LLMs in Russian and English. Participants and LLMs were shown colors and were asked to choose from the same 15-color list. The count of chosen colors for each option is given in parentheses. The color of a response cluster in the maps represents its average color (see Supplementary Fig. S3 for all maps). Colors for which less than 50% and 90% of the times the dominant color term was selected were indicated by “−” and “*”, respectively. If the dominant color term was selected more than 90% of the time, no marking was used.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists are queued

They are fetched later and appear here then.

References 44 · Cited by 73

Research works printed but not linked: 7

  1. Forster, M. & von Herder, J.G. In The Stanford Encyclopedia of Philosophy (Zalta, E. N. & Nodelman, U. eds.). Winter 2023 Ed. (Metaphysics Research Lab, Stanford University, 2023).
  2. Patel, R. & Pavlick, E. Mapping language models to grounded conceptual spaces. In International Conference on Learning Representations (2021).
  3. Marjieh, R. et al. Words are all you need? capturing human sensory similarity with textual descriptors. In The Eleventh International Conference on Learning Representations (2022).
  4. Berlin, B. & Kay, P. Basic Color Terms: Their Universality and Evolution (University of California Press, 1991).
  5. Tucker, M., Levy, R., Shah, J. A. & Zaslavsky, N. Trading off utility, informativeness, and complexity in emergent communication. Adv. Neural Inf. Process. Syst. 35, 22214–22228 (2022).
  6. Harrison, P. et al. Gibbs sampling with people. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F. & Lin, H. (eds.) Advances in Neural Information Processing Systems. Vol. 33. 10659–10671 (Curran Associates, Inc., 2020).
  7. Brown, W. Some experimental results in the correlation of mental abilities 1. Br. J. Psychol. 1904–1920(3), 296–322 (1910).

Metadata

Authors
Marjieh, R., Sucholutsky, I., van Rijn, P., Jacoby, N., & Griffiths, T. L.
Affiliations
  • Princeton University (US)
  • Cornell University (US)
Corresponding author
Marjieh, Raja
Venue
Scientific Reports
Published
2024-09-13
Volume
14
Issue
1
Pages
21445-21445
ISSN
2045-2322 (ISSN-L)
OpenAlex
W4402512774
Cited by
73
References
44
Citations by year
2026: 24 · 2025: 31 · 2024: 12
FWCI
27.17
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
This research was supported by the NOMIS Foundation, and an NSERC fellowship to author IS (567554-2022)
Topics
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
Keywords
large language models, cross-modal perception, color wheel, language and perception, cross-linguistic variation, color naming, GPT-4, vision-language models, psychophysical data
MeSH terms
Humans, Judgment, Language, Visual Perception

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Large language models predict human sensory judgments across six modalities. https://app.cochat.ai/papers/10.1038/s41598-024-72071-1
The paper
Marjieh, R., Sucholutsky, I., van Rijn, P., Jacoby, N., & Griffiths, T. L. (2024). Large language models predict human sensory judgments across six modalities. Scientific Reports. https://doi.org/10.1038/s41598-024-72071-1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.