The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index; 11 of the 19 sources its claims cite were checked for support.
Rating norms should be calculated from cumulative link mixed effects models
Open accessPublished version of a preprint
Abstract
Studies which provide norms of Likert ratings typically report per-item summary statistics. Traditionally, these summary statistics comprise the mean and the standard deviation (SD) of the ratings, and the number of observations. Such summary statistics can preserve the rank order of items, but provide distorted estimates of the relative distances between items because of the ordinal nature of Likert ratings. Inter-item relations in such ordinal scales can be more appropriately modelled by cumulative link mixed effects models (CLMMs). In a series of simulations, and with a reanalysis of an existing rating norms dataset, we show that CLMMs can be used to more accurately norm items, and can provide summary statistics analogous to the traditionally reported means and SDs, but which are disentangled from participants' response biases. CLMMs can be applied to solve important statistical issues that exist for more traditional analyses of rating norms.
Overview
CoChat Summary
TL;DR
The paper argues that Likert rating norms should be computed from cumulative link mixed models (CLMMs), because raw means and SDs distort inter-item distances when ordinal responses are treated as continuous. Across simulations and a reanalysis of character-similarity ratings, CLMM random effects recovered latent item means and SDs more accurately and were less affected by response patterns.
Key findings
CLMMs recovered item latent means more accurately than raw Likert means across five simulated response patterns (equidistant, left-biased, right-biased, edge-biased, centre-biased), while raw means were distorted by nonlinear response mappings. In Simulation 2, adding participant random effects improved item random-effect accuracy, especially when participants were more variable than items; when items were more variable, there was no gain but also no loss. In Simulation 3, distributional CLMMs recovered latent SDs better than raw SDs; raw SDs underestimated large latent SDs, with items simulated at SD = 8 estimated at only about 1.5–2. In the real character-similarity data, the Bayesian distributional CLMM’s latent mean estimates correlated r = .997 with an equal-variance maximum-likelihood CLMM, and CLMM-derived norms were generally more consistent across split-half samples except at the floor of the scale.
Methods
The paper reports four simulation studies: Simulation 1 simulated 100 items with 25 observations per item; Simulation 2 added 100 participants each rating 25 items; Simulation 2b varied item and participant random-effect SDs from .25 to 5 in steps of .25; Simulation 3 simulated 100 datasets with item and participant random effects for latent mean and SD; Simulation 4 varied the latent distribution or item random effects across normal, logistic, uniform, bimodal, and half-normal distributions. Models were fit in R with the ordinal package or brms/Stan, using probit-link CLMMs; distributional models estimated a discrimination parameter as the inverse latent SD. The real-data application reanalysed Simpson et al. (2013): 2704 character pairs, 677 participants, 81,199 trials, 108–120 trials per participant, and 29–31 ratings per item on a 1–7 similarity scale, using a Bayesian distributional CLMM with six chains of 6000 iterations and thin = 2.
Limitations
The paper states that all simulations and analyses assumed a single response pattern across observations, though it notes that participants or items may show distinct response patterns and that modelling per-participant or per-item thresholds was not evaluated. It also states that Bayesian distributional CLMMs are computationally complex and can take substantially longer to fit, potentially becoming unfeasible for very large datasets; if latent variance is not of interest, simpler equal-variance models may suffice. The paper further notes that subjective judgements remain subjective even after disentangling response patterns, and that floor/ceiling effects can increase uncertainty in latent estimates. Evident from the text: the empirical validation is limited to one reanalysed dataset, and the simulation findings depend on the specified generative models and response-pattern scenarios.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures















Verification
Claims the paper makes
11 Valid · 0 Not valid · 7 Not checked · 109 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“We note that when distributional CLMMs are used to disentangle meaningful and artefactual contributions to items’ SDs, the striking mean–SD relationships identified as problematic by Pollock (2018) are no longer observed.”
§ Rating norms should be calculated from cumulative link mixed effects models > Discussion
What the sources say
- Supportedcites Pollock (2018) source
The Brysbaert et al. (2013) concreteness norms section describes a striking relationship between item means and standard deviations: variability rises in the middle of the rating scale. The General discussion calls the recurring distribution problematic. The statement about what happens after distributional CLMMs are applied is the citing authors’ own observation, not a finding attributed to Pollock. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, Pollock showed that although it was common for participants to agree on ratings at the extremes of the scale (1 and 5), such inter-rater reliability was exceedingly rare for ratings at midpoints in the scale.”
§ Rating norms should be calculated from cumulative link mixed effects models > Introduction
What the sources say
- Supportedcites Pollock (2018) source
In the Brysbaert et al. (2013) concreteness norms section, the discussion of Fig. 2 reports relatively low rating variability at both ends of the 1–5 scale, indicating substantial agreement. It says variability rises in the middle and only a handful of words near 3 have a standard deviation even slightly below 1. (LLM, conf=0.93)
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“The brms package (Bürkner, 2018) for R, an interface to STAN (STAN Development Team, 2021), provides an accessible solution to fit such models.”
§ Rating norms should be calculated from cumulative link mixed effects models > Introduction
What the sources say
- Supportedcites Bürkner, 2018 source
The Introduction describes brms as an accessible interface to Stan and explains that distributional models can predict the mean and other response-distribution parameters simultaneously. Example 2 fits a brms model in which both the location and scale parameters vary. The paper calls this a response distribution rather than a latent distribution, but supports the modeling capability attributed to brms. (LLM, conf=0.90)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“However, such dichotomisation will result in an unnecessary loss of statistical power and precision if continuous alternatives are available (MacCallum et al., 2002; Royston et al., 2006).”
§ Rating norms should be calculated from cumulative link mixed effects models > Introduction
What the sources say
- Supportedcites MacCallum et al., 2002 source
The Summary and Conclusions (p. 20) identify loss of information, effect size, power, and measurement reliability as consequences of dichotomization in most circumstances. They also say these consequences can be avoided by analyzing the original, undichotomized measures, supporting the claim when continuous alternatives are available. (LLM, conf=0.90)
- Not checked — no full text of the sourcecites Royston et al., 2006 source
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We note, however, that modelling data in such cases may be made more tractable with statistical approaches like that outlined by Selker et al. (2019), which allows an arbitrary number of thresholds to be estimated via just two parameters per participant.”
§ Rating norms should be calculated from cumulative link mixed effects models > Discussion
What the sources say
- Supportedcites Selker et al. (2019) source
The Abstract says the model estimates any number of thresholds using two parameters. The Thresholds and Hierarchical extension sections describe individual differences in threshold placement and fitting the model to individual observers. The thresholds are constrained by a linear transformation, rather than freely placed. (LLM, conf=0.92)
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
69 more claims · 3 with their cited sources checked
Create a free account to see all 127 claims and to verify the 1 reference still open.
- Probably not a substantial claim. 26
- It may state two separate claims as one. 34
- The quote only partly states this claim. 1
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Fitting Linear Mixed-Effects Models Using lme4
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015) · Journal of Statistical Software · doi:10.18637/jss.v067.i01 · Cited by 86,835 · cited 1×
“The LMMs were fit via the lme4 package for R (Bates et al., 2015), with a Gaussian identity link for comparability to the raw means.”
MethodRegression Models for Ordinal Data
McCullagh, P. (1980) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1980.tb01109.x · Cited by 4,453 · cited 2×
“This is the approach implemented in cumulative link models (CLMs), where ordinal dependent variables are mapped onto ordered regions of a latent distribution (Bürkner & Vuorre, 2019; McCullagh, 1980).”
MethodOn the practice of dichotomization of quantitative variables.
MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.19 · Cited by 3,538 · cited 1×
“However, such dichotomisation will result in an unnecessary loss of statistical power and precision if continuous alternatives are available (MacCallum et al., 2002; Royston et al., 2006).”
BackgroundAdvanced Bayesian Multilevel Modeling with the R Package brms
Bürkner, P.-C. (2018) · The R Journal · doi:10.32614/rj-2018-017 · Cited by 3,015
Dichotomizing continuous predictors in multiple regression: a bad idea
Royston, P., Altman, D. G., & Sauerbrei, W. (2006) · Statistics in Medicine · doi:10.1002/sim.2331 · Cited by 2,312
Norms of valence, arousal, and dominance for 13,915 English lemmas
Warriner, A. B., Kuperman, V., & Brysbaert, M. (2013) · Behavior Research Methods · doi:10.3758/s13428-012-0314-x · Cited by 1,959
Concreteness ratings for 40 thousand generally known English word lemmas
Brysbaert, M., Warriner, A. B., & Kuperman, V. (2014) · Behavior Research Methods · doi:10.3758/s13428-013-0403-5 · Cited by 1,821
The Chicago face database: a free stimulus set of faces and norming data
Ma, D. S., Correll, J., & Wittenbrink, B. (2015) · Behavior Research Methods · doi:10.3758/s13428-014-0532-5 · Cited by 1,788 · cited 1×
“, 2014) or faces (Ma et al., 2015), and melodies (Belfi & Kacirek, 2021).”
Response Styles in Marketing Research: A Cross-National Investigation
Baumgartner, H., & Steenkamp, J.-B. E. M. (2001) · Journal of Marketing Research · doi:10.1509/jmkr.38.2.143.18840 · Cited by 1,255
Cited by
Waisman, T., Williams, Z. J., Cage, E., Santhanam, S. P., Magiati, I., Dwyer, P. S. R., Stockwell, K. M., Kofner, B., Brown, H. M., Davidson, D. Z., et al. (2022) · Autism · doi:10.1177/13623613221097207 · Cited by 81
Rousselet, G. A., Pernet, C. R., & Wilcox, R. R. (2023) · Meta-Psychology · doi:10.15626/mp.2019.2058 · Cited by 119
“…not require the bootstrap: for instance diffusion models of reaction times (Matzke & Wagenmakers, 2009; Voss et al., 2013), beta-binomial models of accuracy scores (Kr-uschke, 2014), and ordinal models of Likert scale results (Bürkner & Vuorre, 2019; Liddell & Kruschke, 2018; Taylor et al., 2022).”
MethodSelf-citationConcreteness ratings for 62,000 English multiword expressions
Muraki, E. J., Abdalla, S., Brysbaert, M., & Pexman, P. M. (2022) · Behavior Research Methods · doi:10.3758/s13428-022-01912-6 · Cited by 61
Emoji Alter the Perception of Emotion in Affectively Neutral Text messages
Neel, L. A. G., McKechnie, J. G., Robus, C. M., & Hand, C. J. (2023) · Journal of Nonverbal Behavior · doi:10.1007/s10919-022-00421-6 · Cited by 59
“, 2018) and individual stimulus items (Taylor et al., 2021).”
BackgroundInfluentialA solution to the pervasive problem of response bias in self-reports
Grimmond, J., Brown, S. D., & Hawkins, G. E. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2412807122 · Cited by 44
Metadata
- Authors
- Taylor, J. E., Rousselet, G. A., Scheepers, C., & Sereno, S. C.
- Affiliations
- University of Glasgow (GB)
- Corresponding author
- Taylor, Jack E.
- Venue
- Behavior Research Methods
- Published
- 2022-09-14
- Volume
- 55
- Issue
- 5
- Pages
- 2175-2196
- ISSN
- 1554-351X (ISSN-L)
- 1554-3528
- PMC
- PMC10439063
- OpenAlex
- W4295681625
- Cited by
- 126
- References
- 42
- Citations by year
- 2026: 44 · 2025: 37 · 2024: 26 · 2023: 13 · 2022: 6
- FWCI
- 12.55
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Behavior Research Methods (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion, cc-by-sa)
- OSF Preprints (OSF Preprints) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion)
- Funders
- Economic and Social Research Council
- Topics
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
- Keywords
- Likert scale, ordinal data, response bias, summary statistics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Rating norms should be calculated from cumulative link mixed effects models. https://app.cochat.ai/papers/10.3758/s13428-022-01814-7
- The paper
- Taylor, J. E., Rousselet, G. A., Scheepers, C., & Sereno, S. C. (2022). Rating norms should be calculated from cumulative link mixed effects models. Behavior Research Methods. https://doi.org/10.3758/s13428-022-01814-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.