CoChat

Statistical and methodological problems with concreteness and other semantic variables: A list memory experiment case study

Open access
Pollock, Lewis · 2018 · Behavior Research Methods

The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.

Abstract

The purpose of this article is to highlight problems with a range of semantic psycholinguistic variables (concreteness, imageability, individual modality norms, and emotional valence) and to provide a way of avoiding these problems. Focusing on concreteness, I show that for a large class of words in the Brysbaert, Warriner, and Kuperman (Behavior Research Methods 46: 904-911, 2013) concreteness norms, the mean concreteness values do not reflect the judgments that actual participants made. This problem applies to nearly every word in the middle of the concreteness scale. Using list memory experiments as a case study, I show that many of the "abstract" stimuli in concreteness experiments are not unequivocally abstract. Instead, they are simply those words about which participants tend to disagree. I report three replications of list memory experiments in which the contrast between concrete and abstract stimuli was maximized, so that the mean concreteness values were accurate reflections of participants' judgments. The first two experiments did not produce a concreteness effect. After I introduced an additional control, the third experiment did produce a concreteness effect. The article closes with a discussion of the implications of these results, as well as a consideration of variables other than concreteness. The sensorimotor experience variables (imageability and individual modality norms) show the same distribution as concreteness. The distribution of emotional valence scores is healthier, but variability in ratings takes on a special significance for this measure because of how the scale is constructed. I recommend that researchers using these variables keep the standard deviations of the ratings of their stimuli as low as possible.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3758/s13428-017-0938-y

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Theoretically possible locations for words rated between 1 and 5 by 25 different participants
Fig. 1 Theoretically possible locations for words rated between 1 and 5 by 25 different participantsCC BY · PubMed Central
Fig. 2 Theoretical versus actual locations
Fig. 2 Theoretical versus actual locationsCC BY · PubMed Central
Fig. 3 Romani et al. (2008) stimuli
Fig. 3 Romani et al. (2008) stimuliCC BY · PubMed Central
Fig. 4 Allen and Hulme (2006) stimuli
Fig. 4 Allen and Hulme (2006) stimuliCC BY · PubMed Central
Fig. 5 Miller and Roodenrys (2009) stimuli
Fig. 5 Miller and Roodenrys (2009) stimuliCC BY · PubMed Central
Fig. 6 Walker and Hulme (1999) stimuli
Fig. 6 Walker and Hulme (1999) stimuliCC BY · PubMed Central
Fig. 7 Concrete and abstract stimuli featured in Experiment 1
Fig. 7 Concrete and abstract stimuli featured in Experiment 1CC BY · PubMed Central
Fig. 8 Concrete and abstract stimuli featured in Experiment 2
Fig. 8 Concrete and abstract stimuli featured in Experiment 2CC BY · PubMed Central
Fig. 9 Means and standard deviations of imageability ratings for 6,000 words (Cortese & Fugett, 2004; Schock et al., 2012)
Fig. 9 Means and standard deviations of imageability ratings for 6,000 words (Cortese & Fugett, 2004; Schock et al., 2012)CC BY · PubMed Central
Fig. 10 Means and standard deviations of Lynott and Connell’s (2012) modality exclusivity norms
Fig. 10 Means and standard deviations of Lynott and Connell’s (2012) modality exclusivity normsCC BY · PubMed Central
Fig. 11 Means and standard deviations of Warriner et al.’s (2013) emotional valence norms
Fig. 11 Means and standard deviations of Warriner et al.’s (2013) emotional valence normsCC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 42 references

Cited by

See all 94

Metadata

Authors
Pollock, L.
Affiliations
University College London (GB)
Corresponding author
Pollock, Lewis
Venue
Behavior Research Methods
Published
2017-07-13
Volume
50
Issue
3
Pages
1198-1216
ISSN
  • 1554-351X (ISSN-L)
  • 1554-3528
OpenAlex
W2734627439
Cited by
90
References
42
Citations by year
2026: 11 · 2025: 10 · 2024: 10 · 2023: 14 · 2022: 12 · 2021: 14 · 2020: 9 · 2019: 5 · 2018: 4 · 2017: 1
FWCI
2.98
Type
journal article
Language
en
Open access
Locations
  • Behavior Research Methods (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, other-oa)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • UCL Discovery (University College London) (submittedVersion)
Funders
Economic and Social Research Council (ES/J500185/1)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
Keywords
concreteness, semantic variables, imageability, emotional valence, concreteness effect, abstract words, sensorimotor experience
MeSH terms
Data Collection, Emotions, Female, Humans, Judgment, Male, Memory, Psycholinguistics, Research Design, Semantics, Behavioral Research, Young Adult

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Statistical and methodological problems with concreteness and other semantic variables: A list memory experiment case study. https://app.cochat.ai/papers/10.3758/s13428-017-0938-y
The paper
Pollock, L. (2018). Statistical and methodological problems with concreteness and other semantic variables: A list memory experiment case study. Behavior Research Methods. https://doi.org/10.3758/s13428-017-0938-y

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.