Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index; 11 of the 19 sources its claims cite were checked for support.

Rating norms should be calculated from cumulative link mixed effects models

Open access

Published version of a preprint

Abstract

Studies which provide norms of Likert ratings typically report per-item summary statistics. Traditionally, these summary statistics comprise the mean and the standard deviation (SD) of the ratings, and the number of observations. Such summary statistics can preserve the rank order of items, but provide distorted estimates of the relative distances between items because of the ordinal nature of Likert ratings. Inter-item relations in such ordinal scales can be more appropriately modelled by cumulative link mixed effects models (CLMMs). In a series of simulations, and with a reanalysis of an existing rating norms dataset, we show that CLMMs can be used to more accurately norm items, and can provide summary statistics analogous to the traditionally reported means and SDs, but which are disentangled from participants' response biases. CLMMs can be applied to solve important statistical issues that exist for more traditional analyses of rating norms.

Overview

CoChat Summary

TL;DR

The paper argues that Likert rating norms should be computed from cumulative link mixed models (CLMMs), because raw means and SDs distort inter-item distances when ordinal responses are treated as continuous. Across simulations and a reanalysis of character-similarity ratings, CLMM random effects recovered latent item means and SDs more accurately and were less affected by response patterns.

Key findings

CLMMs recovered item latent means more accurately than raw Likert means across five simulated response patterns (equidistant, left-biased, right-biased, edge-biased, centre-biased), while raw means were distorted by nonlinear response mappings. In Simulation 2, adding participant random effects improved item random-effect accuracy, especially when participants were more variable than items; when items were more variable, there was no gain but also no loss. In Simulation 3, distributional CLMMs recovered latent SDs better than raw SDs; raw SDs underestimated large latent SDs, with items simulated at SD = 8 estimated at only about 1.5–2. In the real character-similarity data, the Bayesian distributional CLMM’s latent mean estimates correlated r = .997 with an equal-variance maximum-likelihood CLMM, and CLMM-derived norms were generally more consistent across split-half samples except at the floor of the scale.

Methods

The paper reports four simulation studies: Simulation 1 simulated 100 items with 25 observations per item; Simulation 2 added 100 participants each rating 25 items; Simulation 2b varied item and participant random-effect SDs from .25 to 5 in steps of .25; Simulation 3 simulated 100 datasets with item and participant random effects for latent mean and SD; Simulation 4 varied the latent distribution or item random effects across normal, logistic, uniform, bimodal, and half-normal distributions. Models were fit in R with the ordinal package or brms/Stan, using probit-link CLMMs; distributional models estimated a discrimination parameter as the inverse latent SD. The real-data application reanalysed Simpson et al. (2013): 2704 character pairs, 677 participants, 81,199 trials, 108–120 trials per participant, and 29–31 ratings per item on a 1–7 similarity scale, using a Bayesian distributional CLMM with six chains of 6000 iterations and thin = 2.

Limitations

The paper states that all simulations and analyses assumed a single response pattern across observations, though it notes that participants or items may show distinct response patterns and that modelling per-participant or per-item thresholds was not evaluated. It also states that Bayesian distributional CLMMs are computationally complex and can take substantially longer to fit, potentially becoming unfeasible for very large datasets; if latent variance is not of interest, simpler equal-variance models may suffice. The paper further notes that subjective judgements remain subjective even after disentangling response patterns, and that floor/ceiling effects can increase uncertainty in latent estimates. Evident from the text: the empirical validation is limited to one reanalysed dataset, and the simulation findings depend on the specified generative models and response-pattern scenarios.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3758/s13428-022-01814-7

Cites

42 of 42 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 The assumed relationship between a continuous latent distribution and ordinal Likert responses (here, on a 1–5 scale). Each Likert response corresponds to a region of the latent distribution, highlighted in corresponding colours. The probability of observing any given Likert response is the probability of a value being drawn from the latent distribution which is between the lower and upper bounds of that Likert response’s region. In the example illustrated, the latent distribution is assumed to be normal (as is the case for a probit-link function). The nonlinearities in this example response pattern mean that the most likely response would be 2, while the responses of 1 and 5 would be comparably rare. This response pattern would therefore bias means of observations towards the Likert response 2, and away from the scale’s extremities
Fig. 1 The assumed relationship between a continuous latent distribution and ordinal Likert responses (here, on a 1–5 scale). Each Likert response corresponds to a region of the latent distribution, highlighted in corresponding colours. The probability of observing any given Likert response is the probability of a value being drawn from the latent distribution which is between the lower and upper bounds of that Likert response’s region. In the example illustrated, the latent distribution is assumed to be normal (as is the case for a probit-link function). The nonlinearities in this example response pattern mean that the most likely response would be 2, while the responses of 1 and 5 would be comparably rare. This response pattern would therefore bias means of observations towards the Likert response 2, and away from the scale’s extremitiesCC BY · PubMed Central
Fig. 2 The relationship between the mean and SD of items’ Likert ratings (1–5 scale) in word concreteness, from Brysbaert et al. (2014). The pattern suggests that responses are most consistent at the extremes of the Likert scale, but that items with averages at the midpoints of the Likert scale elicit less consistent responses
Fig. 2 The relationship between the mean and SD of items’ Likert ratings (1–5 scale) in word concreteness, from Brysbaert et al. (2014). The pattern suggests that responses are most consistent at the extremes of the Likert scale, but that items with averages at the midpoints of the Likert scale elicit less consistent responsesCC BY · PubMed Central
Fig. 3 Mean–SD relationships for judgements of words on three semantic variables in the Glasgow Norms (Scott et al., 2019). Dominance was judged on a 1–9 Likert scale, while Familiarity and Gender were judged on 1–7 scales
Fig. 3 Mean–SD relationships for judgements of words on three semantic variables in the Glasgow Norms (Scott et al., 2019). Dominance was judged on a 1–9 Likert scale, while Familiarity and Gender were judged on 1–7 scalesCC BY · PubMed Central
Fig. 4 Illustration of how response patterns affect Likert responses. Given the same latent distribution, the five example response patterns we used in the simulations alter the probability of different Likert responses. The top half of the figure, a, shows the locations of the thresholds for each response pattern, highlighting how changes in the latent mean alter the proportion of the latent distribution which maps onto each Likert response. Importantly, this effect differs among response patterns. To illustrate this point, for each response pattern, the densities of three example distributions (white curves) are shown, with means of –2.5, 0, and 2.5, and an identical SD of 1. An observation sampled from one of these distributions would fall into one coloured region and would be mapped onto the corresponding Likert response. The bottom half of the figure, b, shows how the change in the mean of the latent distribution (on the x-axis) alters the probability (cumulative percentage; y-axis) of observing any Likert response differentially in each of the five response patterns for an identical latent distribution of mean 0 and SD 1. The same example three distributions as in panel a are highlighted with white vertical lines
Fig. 4 Illustration of how response patterns affect Likert responses. Given the same latent distribution, the five example response patterns we used in the simulations alter the probability of different Likert responses. The top half of the figure, a, shows the locations of the thresholds for each response pattern, highlighting how changes in the latent mean alter the proportion of the latent distribution which maps onto each Likert response. Importantly, this effect differs among response patterns. To illustrate this point, for each response pattern, the densities of three example distributions (white curves) are shown, with means of –2.5, 0, and 2.5, and an identical SD of 1. An observation sampled from one of these distributions would fall into one coloured region and would be mapped onto the corresponding Likert response. The bottom half of the figure, b, shows how the change in the mean of the latent distribution (on the x-axis) alters the probability (cumulative percentage; y-axis) of observing any Likert response differentially in each of the five response patterns for an identical latent distribution of mean 0 and SD 1. The same example three distributions as in panel a are highlighted with white vertical linesCC BY · PubMed Central
Fig. 5 Results of Simulation 1: CLMMs recover items’ latent distribution random effects from the five example response patterns. The panels show: a the relationship between means and SDs of ratings, b the relationship between items’ simulated latent means and their mean ratings, and c the relationship between items’ simulated latent means and estimated random effects from the CLMM. The relationships in panels b and c shown with the black lines were estimated via locally estimated scatterplot smoothing (LOESS), with a span parameter of .75. The dashed red lines show an expected linear relationship for reference, identical across all response patterns. In all panels, results from all simulation iterations are concatenated. The results show that the relationship between items’ means and SDs of ratings differs markedly between simulated response patterns, even though the simulated values in the latent distribution were identical. While averaging over ordinal responses works well when the responses are generated from equidistant thresholds, any other response pattern leads to nonlinear inaccuracies in the values. CLMMs, meanwhile, account for any pattern of thresholds and more accurately recover the items’ distributions in the latent variable
Fig. 5 Results of Simulation 1: CLMMs recover items’ latent distribution random effects from the five example response patterns. The panels show: a the relationship between means and SDs of ratings, b the relationship between items’ simulated latent means and their mean ratings, and c the relationship between items’ simulated latent means and estimated random effects from the CLMM. The relationships in panels b and c shown with the black lines were estimated via locally estimated scatterplot smoothing (LOESS), with a span parameter of .75. The dashed red lines show an expected linear relationship for reference, identical across all response patterns. In all panels, results from all simulation iterations are concatenated. The results show that the relationship between items’ means and SDs of ratings differs markedly between simulated response patterns, even though the simulated values in the latent distribution were identical. While averaging over ordinal responses works well when the responses are generated from equidistant thresholds, any other response pattern leads to nonlinear inaccuracies in the values. CLMMs, meanwhile, account for any pattern of thresholds and more accurately recover the items’ distributions in the latent variableCC BY · PubMed Central
Fig. 6 Results of Simulation 2: CLMMs recover items’ latent distribution random effects when per-participant random intercepts are also simulated. Panels a and b show the same information about items as the respective panels in Fig. 5, but for data which additionally simulated participant random intercepts. The additional variability from participant random effects has led to smoother patterns in panel a. Panel c shows the relationship between simulated latent distribution values and item random effect estimates, from the CLMM estimating item random intercepts only (green), and from the CLMM estimating both item and participant random intercepts (orange). As in Fig. 5, each line represents a LOESS estimate (span parameter of .75) of the relationship. Panel d shows the density of the error in the items’ random effect estimates (i.e., error = simulated value − estimated value) from both types of CLMM estimated. Here, observations from all 100 simulation iterations are concatenated. The distribution of the errors shows that including participant random effects improves the accuracy of estimates for item random effects. In panel e, this same difference is again presented, though the random effect estimates have been scaled by standard deviation to account for the differences in estimated magnitude shown in panel c
Fig. 6 Results of Simulation 2: CLMMs recover items’ latent distribution random effects when per-participant random intercepts are also simulated. Panels a and b show the same information about items as the respective panels in Fig. 5, but for data which additionally simulated participant random intercepts. The additional variability from participant random effects has led to smoother patterns in panel a. Panel c shows the relationship between simulated latent distribution values and item random effect estimates, from the CLMM estimating item random intercepts only (green), and from the CLMM estimating both item and participant random intercepts (orange). As in Fig. 5, each line represents a LOESS estimate (span parameter of .75) of the relationship. Panel d shows the density of the error in the items’ random effect estimates (i.e., error = simulated value − estimated value) from both types of CLMM estimated. Here, observations from all 100 simulation iterations are concatenated. The distribution of the errors shows that including participant random effects improves the accuracy of estimates for item random effects. In panel e, this same difference is again presented, though the random effect estimates have been scaled by standard deviation to account for the differences in estimated magnitude shown in panel cCC BY · PubMed Central
Fig. 7 Results of Simulation 2b: effect of varying the magnitude of item (x-axis) and participant (y-axis) random effects on estimation accuracy of item random effects. Panel a shows the estimated SDs of errors of item random effects, where estimates are scaled (as in Fig. 6e) to account for differences in magnitude. Estimates in panel a are shown separately for a model estimating only item random effects (left), and a model estimating both item and participant random effects (right). Panel b shows the difference between the estimates from the two models, calculated as item estimates from the less complex model (estimating only item random effects) minus those of the more complex model (estimating both item and participant random effects). Values in panel b therefore index the reduction in error that results from accounting for participant random effects (e.g., a value of .3 reflects a reduction of .3 SDs in the magnitude of errors)
Fig. 7 Results of Simulation 2b: effect of varying the magnitude of item (x-axis) and participant (y-axis) random effects on estimation accuracy of item random effects. Panel a shows the estimated SDs of errors of item random effects, where estimates are scaled (as in Fig. 6e) to account for differences in magnitude. Estimates in panel a are shown separately for a model estimating only item random effects (left), and a model estimating both item and participant random effects (right). Panel b shows the difference between the estimates from the two models, calculated as item estimates from the less complex model (estimating only item random effects) minus those of the more complex model (estimating both item and participant random effects). Values in panel b therefore index the reduction in error that results from accounting for participant random effects (e.g., a value of .3 reflects a reduction of .3 SDs in the magnitude of errors)CC BY · PubMed Central
Fig. 8 Results of Simulation 3: efficacy of distributional unequal variance CLMMs for calculating the SD of latent variables’ variance from Likert response data. Panels a, b, and c show that the findings from the previous simulation also apply to models which calculate differences in both the mean and variance of the latent distribution. Panel d shows the relationship between items’ simulated latent variable SD, and the SD of Likert ratings, while panel e shows that differences in the latent distribution’s SD are more accurately retrieved by random effects in the disc parameter. Lines tracking relationships in panels b–e are estimated via LOESS (span parameter = .75)
Fig. 8 Results of Simulation 3: efficacy of distributional unequal variance CLMMs for calculating the SD of latent variables’ variance from Likert response data. Panels a, b, and c show that the findings from the previous simulation also apply to models which calculate differences in both the mean and variance of the latent distribution. Panel d shows the relationship between items’ simulated latent variable SD, and the SD of Likert ratings, while panel e shows that differences in the latent distribution’s SD are more accurately retrieved by random effects in the disc parameter. Lines tracking relationships in panels b–e are estimated via LOESS (span parameter = .75)CC BY · PubMed Central
Fig. 9 Comparison between a CLMM assuming equal variance across observations (blue), and a CLMM estimating differences in the variance of the latent distribution (yellow). Panel a shows that a similar pattern exists for both models between simulated latent means, and those estimated by the models’ random effects (scaled for comparability between the models). The estimates are so similar that results for the equal variances model are largely overlaid by results from the distributional model. Panel b shows the density of the differences between simulated latent mean values and scaled estimates from CLMM random effects. While estimating items’ differences in latent variance may provide a small gain in accuracy for the estimated means, reflected by the slightly heavier tails in the density plots, this improvement is minimal for data simulated here
Fig. 9 Comparison between a CLMM assuming equal variance across observations (blue), and a CLMM estimating differences in the variance of the latent distribution (yellow). Panel a shows that a similar pattern exists for both models between simulated latent means, and those estimated by the models’ random effects (scaled for comparability between the models). The estimates are so similar that results for the equal variances model are largely overlaid by results from the distributional model. Panel b shows the density of the differences between simulated latent mean values and scaled estimates from CLMM random effects. While estimating items’ differences in latent variance may provide a small gain in accuracy for the estimated means, reflected by the slightly heavier tails in the density plots, this improvement is minimal for data simulated hereCC BY · PubMed Central
Fig. 10 Results of Simulation 4a: varying the shape of the latent distribution. Across all non-normal distributions (panel a), a similar heterogenous pattern in the mean–SD relationship in Likert ratings was observed, though it is notably asymmetrical in the case of the half-normal distribution (panel b). For all distributions, estimates of items’ simulated latent variable values were distorted by the edge-biased response pattern when estimated via the mean of simulated Likert ratings (panel c). In contrast, random effects estimates from the CLMM more accurately retrieved the simulated latent variable values, with similar accuracy across the distribution (panel d)
Fig. 10 Results of Simulation 4a: varying the shape of the latent distribution. Across all non-normal distributions (panel a), a similar heterogenous pattern in the mean–SD relationship in Likert ratings was observed, though it is notably asymmetrical in the case of the half-normal distribution (panel b). For all distributions, estimates of items’ simulated latent variable values were distorted by the edge-biased response pattern when estimated via the mean of simulated Likert ratings (panel c). In contrast, random effects estimates from the CLMM more accurately retrieved the simulated latent variable values, with similar accuracy across the distribution (panel d)CC BY · PubMed Central
Fig. 11 Results of Simulation 4b: varying the distribution of the item random effects. Across all non-normal distributions (panel a), the mean–SD relationship for Likert ratings (panel b) showed greatest inconsistency at the midpoints of the Likert scale, although this reflects any asymmetries in the random effect distribution, as is shown for the half-normal distribution. For all non-normal random effect distributions simulated, estimates from averages of Likert ratings (panel c) are distorted by the response pattern, and are less accurate than random effect estimates from CLMMs (panel d), which scale more linearly with simulated values. We note that the unusual pattern in panel c for random effects drawn from a logistic distribution does not reflect the pattern of observations well, but is an artefact of the LOESS method of estimation
Fig. 11 Results of Simulation 4b: varying the distribution of the item random effects. Across all non-normal distributions (panel a), the mean–SD relationship for Likert ratings (panel b) showed greatest inconsistency at the midpoints of the Likert scale, although this reflects any asymmetries in the random effect distribution, as is shown for the half-normal distribution. For all non-normal random effect distributions simulated, estimates from averages of Likert ratings (panel c) are distorted by the response pattern, and are less accurate than random effect estimates from CLMMs (panel d), which scale more linearly with simulated values. We note that the unusual pattern in panel c for random effects drawn from a logistic distribution does not reflect the pattern of observations well, but is an artefact of the LOESS method of estimationCC BY · PubMed Central
Fig. 12 Modelling of character similarity ratings from Simpson et al. (2013): Item random effect results of the Bayesian distributional CLMM. Six pairs of Arial characters are highlighted as examples, ranging from o-j at a low level of similarity, to u-ù at a high level of similarity. Panels depict: a the mean–SD relationship in items’ ratings; b the mean-SD relationship in the latent distribution; c the relationship between each item’s mean rating and its latent distribution mean as estimated by the CLMM; d the relationship between ratings’ SDs and the estimated latent SDs; and e the predicted densities of the latent distributions for the six example items, calculated from their random effect estimates for the mu and disc parameters. Coloured regions indicate the mapping from latent values to Likert responses, where the boundaries between coloured regions reflect the estimated locations of the latent thresholds
Fig. 12 Modelling of character similarity ratings from Simpson et al. (2013): Item random effect results of the Bayesian distributional CLMM. Six pairs of Arial characters are highlighted as examples, ranging from o-j at a low level of similarity, to u-ù at a high level of similarity. Panels depict: a the mean–SD relationship in items’ ratings; b the mean-SD relationship in the latent distribution; c the relationship between each item’s mean rating and its latent distribution mean as estimated by the CLMM; d the relationship between ratings’ SDs and the estimated latent SDs; and e the predicted densities of the latent distributions for the six example items, calculated from their random effect estimates for the mu and disc parameters. Coloured regions indicate the mapping from latent values to Likert responses, where the boundaries between coloured regions reflect the estimated locations of the latent thresholdsCC BY · PubMed Central
Fig. 13 Uncertainty in the estimates of example pairs of characters’ means and standard deviations in the latent distribution. Ellipsoids present the 50, 75, and 89% highest density intervals (HDIs) in the posterior samples, while points show the median estimates for each pair of characters
Fig. 13 Uncertainty in the estimates of example pairs of characters’ means and standard deviations in the latent distribution. Ellipsoids present the 50, 75, and 89% highest density intervals (HDIs) in the posterior samples, while points show the median estimates for each pair of charactersCC BY · PubMed Central
Fig. 14 The latent mean estimates from the Bayesian distributional CLMM, fit with the brms package, correlate very highly with estimates from the model assuming equal variance, fit with the ordinal package
Fig. 14 The latent mean estimates from the Bayesian distributional CLMM, fit with the brms package, correlate very highly with estimates from the model assuming equal variance, fit with the ordinal packageCC BY · PubMed Central
Fig. 15 Norm consistency. Results of the analysis examining consistency in norms estimates for three approaches: raw means, LMM random effects, and CLMM random effects. Panel a depicts the test–retest consistency for 30 example items, from one test–retest iteration. Estimates for the same items, from the three different approaches, are joined by grey lines, while the dashed diagonal line depicts perfect consistency between the two samples for reference. The lower region of panel b shows how uncertainty in latent mean estimates (width of the 89% HDI) from the Bayesian distributional model varies across the latent scale. The coloured bands depict the 89% HDIs for the estimated threshold locations (e.g., 1|2 is the threshold between ratings of 1 and 2). The upper region of panel b shows the distribution of differences between samples A and B, with observations combined from 100 iterations of the test–retest procedure. The differences are depicted separately for items which were either below (left) or above (right) the lowest threshold. The jagged appearance of the density plot of differences for raw means reflects that there is only a finite number of possible values the average rating can take without the additional discriminatory power provided by random effects
Fig. 15 Norm consistency. Results of the analysis examining consistency in norms estimates for three approaches: raw means, LMM random effects, and CLMM random effects. Panel a depicts the test–retest consistency for 30 example items, from one test–retest iteration. Estimates for the same items, from the three different approaches, are joined by grey lines, while the dashed diagonal line depicts perfect consistency between the two samples for reference. The lower region of panel b shows how uncertainty in latent mean estimates (width of the 89% HDI) from the Bayesian distributional model varies across the latent scale. The coloured bands depict the 89% HDIs for the estimated threshold locations (e.g., 1|2 is the threshold between ratings of 1 and 2). The upper region of panel b shows the distribution of differences between samples A and B, with observations combined from 100 iterations of the test–retest procedure. The differences are depicted separately for items which were either below (left) or above (right) the lowest threshold. The jagged appearance of the density plot of differences for raw means reflects that there is only a finite number of possible values the average rating can take without the additional discriminatory power provided by random effectsCC BY · PubMed Central

Verification

Claims the paper makes

11 Valid · 0 Not valid · 7 Not checked · 109 Own results and interpretations

69 more claims · 3 with their cited sources checked

Create a free account to see all 127 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 26
  • It may state two separate claims as one. 34
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Citations

Cites

Show all 42 references

Cited by

See all 126

Metadata

Authors
Taylor, J. E., Rousselet, G. A., Scheepers, C., & Sereno, S. C.
Affiliations
University of Glasgow (GB)
Corresponding author
Taylor, Jack E.
Venue
Behavior Research Methods
Published
2022-09-14
Volume
55
Issue
5
Pages
2175-2196
ISSN
  • 1554-351X (ISSN-L)
  • 1554-3528
OpenAlex
W4295681625
Cited by
126
References
42
Citations by year
2026: 44 · 2025: 37 · 2024: 26 · 2023: 13 · 2022: 6
FWCI
12.55
Type
journal article
Language
en
Open access
Licence
Locations
  • Behavior Research Methods (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion, cc-by-sa)
  • OSF Preprints (OSF Preprints) (submittedVersion)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
  • PsyArXiv (OSF Preprints) (submittedVersion)
Funders
Economic and Social Research Council
Topics
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Reliability and Agreement in Measurement
Keywords
Likert scale, ordinal data, response bias, summary statistics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Rating norms should be calculated from cumulative link mixed effects models. https://app.cochat.ai/papers/10.3758/s13428-022-01814-7
The paper
Taylor, J. E., Rousselet, G. A., Scheepers, C., & Sereno, S. C. (2022). Rating norms should be calculated from cumulative link mixed effects models. Behavior Research Methods. https://doi.org/10.3758/s13428-022-01814-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.