Statistical and methodological problems with concreteness and other semantic variables: A list memory experiment case study
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
The purpose of this article is to highlight problems with a range of semantic psycholinguistic variables (concreteness, imageability, individual modality norms, and emotional valence) and to provide a way of avoiding these problems. Focusing on concreteness, I show that for a large class of words in the Brysbaert, Warriner, and Kuperman (Behavior Research Methods 46: 904-911, 2013) concreteness norms, the mean concreteness values do not reflect the judgments that actual participants made. This problem applies to nearly every word in the middle of the concreteness scale. Using list memory experiments as a case study, I show that many of the "abstract" stimuli in concreteness experiments are not unequivocally abstract. Instead, they are simply those words about which participants tend to disagree. I report three replications of list memory experiments in which the contrast between concrete and abstract stimuli was maximized, so that the mean concreteness values were accurate reflections of participants' judgments. The first two experiments did not produce a concreteness effect. After I introduced an additional control, the third experiment did produce a concreteness effect. The article closes with a discussion of the implications of these results, as well as a consideration of variables other than concreteness. The sensorimotor experience variables (imageability and individual modality norms) show the same distribution as concreteness. The distribution of emotional valence scores is healthier, but variability in ratings takes on a special significance for this measure because of how the scale is constructed. I recommend that researchers using these variables keep the standard deviations of the ratings of their stimuli as low as possible.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures











Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“We note that when distributional CLMMs are used to disentangle meaningful and artefactual contributions to items’ SDs, the striking mean–SD relationships identified as problematic by Pollock (2018) are no longer observed.”
From Rating norms should be calculated from cumulative link mixed effects models (2022) · cites this paper as Pollock (2018)
The cited source says what the claim attributes to it.
In the citing paper
“Notably, Pollock (2018) highlighted a common relationship in ratings of word concreteness (Fig. 2), whereby the lowest SDs are observed at the extremes of a Likert scale, while items towards the centre of the scale show much higher SDs.”
From Rating norms should be calculated from cumulative link mixed effects models (2022) · cites this paper as Pollock (2018)
The cited source says what the claim attributes to it.
In the citing paper
“However, Pollock showed that although it was common for participants to agree on ratings at the extremes of the scale (1 and 5), such inter-rater reliability was exceedingly rare for ratings at midpoints in the scale.”
From Rating norms should be calculated from cumulative link mixed effects models (2022) · cites this paper as Pollock (2018)
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Fitting Linear Mixed-Effects Models Using lme4
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835
lmerTest Package: Tests in Linear Mixed Effects Models
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017) · Journal of Statistical Software · doi:10.18637/jss.v082.i13 · Cited by 24,346
Concreteness, imagery, and meaningfulness values for 925 nouns.
Paivio, A., Yuille, J. C., & Madigan, S. A. (1968) · Journal of Experimental Psychology · doi:10.1037/h0025327 · Cited by 2,827 · cited 2×
“She is told that Bconcrete words are experienced by the senses,^ whereas abstract words are not (Paivio et al., 1968).”
BackgroundBalota, D. A., Yap, M. J., Hutchison, K. A., Cortese, M. J., Kessler, B., Loftis, B., Neely, J. H., Nelson, D. L., Simpson, G. B., & Treiman, R. (2007) · Behavior Research Methods · doi:10.3758/bf03193014 · Cited by 2,742 · cited 1×
“Psycholinguistic variable information was gathered from Brysbaert et al. (2013), Kuperman, Stadthagen-Gonzalez, and Brysbaert (2012), and the English Lexicon Project (Balota et al., 2007).”
MethodThe MRC Psycholinguistic Database
Coltheart, M. (1981) · The Quarterly Journal of Experimental Psychology Section A · doi:10.1080/14640748108400805 · Cited by 2,437
Norms of valence, arousal, and dominance for 13,915 English lemmas
Warriner, A. B., Kuperman, V., & Brysbaert, M. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0314-x · Cited by 1,959 · cited 2×
“…of the concreteness ratings; AoA: Age of acquisition; Zipf frequency: Word frequency in Zipf units; N Syll: Number of syllables; Length: Length of word in letters; BG mean : Mean bigram frequency; Absolute Valence: Absolute value of 5 minus the Warriner et al. (2013) emotional valence score.”
BackgroundConcreteness ratings for 40 thousand generally known English word lemmas
Brysbaert, M., Warriner, A. B., & Kuperman, V. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0403-5 · Cited by 1,821 · cited 18×
“Psycholinguistic variable information was gathered from Brysbaert et al. (2013), Kuperman, Stadthagen-Gonzalez, and Brysbaert (2012), and the English Lexicon Project (Balota et al., 2007).”
BackgroundMethodInfluentialAge-of-acquisition ratings for 30,000 English words
Kuperman, V., Stadthagen-Gonzalez, H., & Brysbaert, M. (2012) · Behavior Research Methods · doi:10.3758/s13428-012-0210-4 · Cited by 1,217 · cited 1×
“Psycholinguistic variable information was gathered from Brysbaert et al. (2013), Kuperman, Stadthagen-Gonzalez, and Brysbaert (2012), and the English Lexicon Project (Balota et al., 2007).”
MethodThe representation of abstract words: Why emotion matters.
Kousta, S.-T., Vigliocco, G., Vinson, D. P., Andrews, M., & Del Campo, E. (2011) · Journal of Experimental Psychology: General · doi:10.1037/a0021446 · Cited by 894
Distinct Brain Systems for Processing Concrete and Abstract Concepts
Binder, J. R., Westbury, C. F., McKiernan, K. A., Possing, E. T., & Medler, D. A. (2005) · Journal of Cognitive Neuroscience · doi:10.1162/0898929054021102 · Cited by 656
Cited by
Gorilla in our midst: An online behavioral experiment builder
Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N., & Evershed, J. K. (2020) · Behavior Research Methods · doi:10.3758/s13428-019-01237-x · Cited by 1,862
Interoception: the forgotten modality in perceptual grounding of abstract and concrete concepts
Connell, L., Lynott, D., & Banks, B. (2018) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2017.0143 · Cited by 206
“However, it was possible that this apparent negative relationship between interoception and concreteness was an artifact of the particular concepts near the midpoint of the concreteness scale, where ratings tend to be noisier than ratings at the extremes because they often reflect participant disagreement or confusion…”
BackgroundThe multifaceted abstract brain
Desai, R. H., Reilly, M., & van Dam, W. (2018) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2017.0122 · Cited by 141
“As Pollock [21] has recently pointed out, concepts with intermediate concreteness ratings tend to have high standard deviations.”
Rating norms should be calculated from cumulative link mixed effects models
Taylor, J. E., Rousselet, G. A., Scheepers, C., & Sereno, S. C. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01814-7 · Cited by 126
“For example, item random effects may be bimodally distributed if there are two distinct categories in the data, such as has been argued to be the case for judgements of concreteness (Pollock, 2018).”
BackgroundIconicity ratings for 14,000+ English words
Winter, B., Lupyan, G., Perry, L. K., Dingemanse, M., & Perlman, M. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02112-6 · Cited by 110
“…languages, including British Sign Language (BSL, Vinson et al., 2008) and American Sign Language (ASL, Caselli et al., 2017), and also in spoken languages, including English (Perry et al., 2015, 2018; Winter et al., 2017), Spanish (Hinojosa et al., 2020), and Japanese (A. L. Thompson et al., 2020).”
Influential
Metadata
- Authors
- Pollock, L.
- Affiliations
- University College London (GB)
- Corresponding author
- Pollock, Lewis
- ORCID
- Pollock, Lewis
- Venue
- Behavior Research Methods
- Published
- 2017-07-13
- Volume
- 50
- Issue
- 3
- Pages
- 1198-1216
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- PMC
- PMC5990559
- OpenAlex
- W2734627439
- Cited by
- 90
- References
- 42
- Citations by year
- 2026: 11 · 2025: 10 · 2024: 10 · 2023: 14 · 2022: 12 · 2021: 14 · 2020: 9 · 2019: 5 · 2018: 4 · 2017: 1
- FWCI
- 2.98
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Behavior Research Methods (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Funders
- Economic and Social Research Council (ES/J500185/1)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
- Keywords
- concreteness, semantic variables, imageability, emotional valence, concreteness effect, abstract words, sensorimotor experience
- MeSH terms
- Data Collection, Emotions, Female, Humans, Judgment, Male, Memory, Psycholinguistics, Research Design, Semantics, Behavioral Research, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Statistical and methodological problems with concreteness and other semantic variables: A list memory experiment case study. https://app.cochat.ai/papers/10.3758/s13428-017-0938-y
- The paper
- Pollock, L. (2018). Statistical and methodological problems with concreteness and other semantic variables: A list memory experiment case study. Behavior Research Methods. https://doi.org/10.3758/s13428-017-0938-y
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.