Ratings are Overrated!
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Are ratings of any use in human-computer interaction and user studies at large? If ratings are of limited use, is there a better alternative for quantitative subjective assessment? Beyond the intrinsic shortcomings of human reporting, there are a number of supplementary limitations and fundamental methodological flaws associated with rating-based questionnaires --- i.e. questionnaires that ask participants to rate their level of agreement with a given statement such as a Likert item. While the effect of these pitfalls has been largely downplayed, recent findings from diverse areas of study question the reliability of using ratings. Rank-based questionnaires --- i.e. questionnaires that ask participants to rank two or more options --- appear as the evident alternative that not only eliminates the core limitations of ratings but also simplifies the use of sound methodologies that yield more reliable models of the underlying reported construct: user emotion, preference, or opinion. This paper solicits recent findings from various disciplines interlinked with psychometrics and offers a quick guide for the use, processing and analysis of rank-based questionnaires for the unique advantages they offer. The paper challenges the traditional state-of-practice in human-computer interaction and psychometrics directly contributing towards a paradigm shift in subjective reporting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Use of Ranks in One-Criterion Variance Analysis
Kruskal, W. H., & Wallis, W. A. (1952) · Journal of the American Statistical Association · doi:10.1080/01621459.1952.10483441 · Cited by 13,912 · cited 3×
“In general, approaches for analyzing ordinal ratings should rely on non-parametric statistics: from simple statistical explorations via Spearman’s correlation, to significance tests via the Mann–Whitney test and the Wilcoxon signed-rank test for paired samples (Wilcoxon, 1945), to the Kruskal–Wallis (Kruskal and Walli…”
MethodLikert scales, levels of measurement, and the “laws” of statistics
Norman, G. (2010) · Advances in Health Sciences Education · doi:10.1007/s10459-010-9222-y · Cited by 4,850 · cited 1×
“Norman (2010) showed empirically in one dataset that Pearson’s correlation (which treats ratings as intervals) is robust enough when compared against Spearman’s rank correlation (which treats ratings as ordinal values).”
MethodOn the Theory of Scales of Measurement
Stevens, S. S. (1946) · Science · doi:10.1126/science.103.2684.677 · Cited by 4,761 · cited 5×
“…are Frontiers in ICT | www.frontiersin.org July 2015 | Volume 2 | Article 132 often erroneously converted to integer numbers violating basic axioms of statistics, which suggest that ordinal values cannot be treated as interval values (Stevens, 1946) since the underlying numerical scale is unknown.”
BackgroundInfluentialWhat are emotions? And how can they be measured?
Scherer, K. R. (2005) · Social Science Information · doi:10.1177/0539018405058216 · Cited by 4,264 · cited 2×
“Popular rating-based questionnaires (see Figure 1A for an example) include the Likert-scale (Likert, 1932), the Geneva Wheel model (Scherer, 2005), the Self-Assessment Frontiers in ICT | www.frontiersin.org July 2015 | Volume 2 | Article 131 Rating: A 5-point Likert item example Rank: A…”
BackgroundMethodA Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings
Friedman, M. (1940) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177731944 · Cited by 2,785
The visual analogue scale: Its use in pain measurement
Langley, G. B., & Sheppeard, H. (1985) · Rheumatology International · doi:10.1007/bf00541514 · Cited by 619 · cited 3×
“…when a questionnaire features ratings as numbers (e.g., see Figure 1A), the scale is still ordinal as the numbers in the instrument are only labels; thus, the underlying numerical scale is still unknown and dependent on the participant (Stevens, 1946; Langley and Sheppeard, 1985; Ovadia, 2004).”
BackgroundRanking vs. Preference: A Comparative Study of Self-reporting
Yannakakis, G. N., & Hallam, J. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24600-5_47 · Cited by 90
Self-citation
Research works printed but not linked: 30
- Agresti, 1992, Analysis of ordinal paired comparison data, J. R. Stat. Soc. Ser. C Appl. Stat., 41, 287
- Bardram, 2013, Designing mobile health technology for bipolar disorder: a field trial of the monarca system, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 2627, 10.1145/2470654.2481364 · DOI · Paper page
- Baveye, 2013, A large video database for computational models of induced emotion, Proc. of Affect. Comput. Intell. Int., 13, 10.1109/ACII.2013.9 · DOI
- Brown, 2013, How irt can solve problems of ipsative data in forced-choice questionnaires, Psychol. Methods, 18, 36, 10.1037/a0030641 · DOI · Paper page
- Bryan, 2014, Isse: an interactive source separation editor, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 257, 10.1145/2556288.2557253 · DOI
- Chan, 1991, Response-order effects in Likert-type scales, Educ. Psychol. Meas., 51, 531, 10.1177/0013164491513002 · DOI · Paper page
- Chen, 2014, Duet: exploring joint interactions on a smart phone and a smart watch, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 159, 10.1145/2556288.2556955 · DOI · Paper page
- Čopič Pucihar, 2014, The use of surrounding visual context in handheld ar: device vs. user perspective rendering, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 197, 10.1145/2556288.2557125 · DOI
- Dhar, 2003, The effect of forced choice on choice, J. Market. Res., 40, 146, 10.1509/jmkr.40.2.146.19229 · DOI · Paper page
- Fürnkranz, 2010, Preference Learning
- Goyal, 2014, Effects of implicit sharing in collaborative analysis, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 129, 10.1145/2556288.2557229 · DOI
- Hernandez, 2014, Under pressure: sensing stress of computer users, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 51, 10.1145/2556288.2557165 · DOI · Paper page
- IJsselsteijn, 2008, The Game Experience Questionnaire: Development of a Self-Report Measure to Assess Player Experiences of Digital Games
- Jamieson, 2004, Likert scales: how to (ab) use them, Med. Educ., 38, 1217, 10.1111/j.1365-2929.2004.02012.x · DOI · Paper page
- Jay, 2007, Modeling the effects of delayed haptic and visual feedback in a collaborative virtual environment, ACM Trans. Comput. Hum. Interact., 14, 1275514, 10.1145/1275511 · doi:10.1145/1275511
- Kendall, 1938, A new measure of rank correlation, Biometrika, 81, 10.1093/biomet/30.1-2.81 · DOI · Paper page
- Likert, 1932, A technique for the measurement of attitudes, Arch. Psychol., 140, 1
- Linn, 2000, Measurement and Assessment in Teaching
- Mark, 2014, Stress and multitasking in everyday college life: an empirical study of online activity, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 41, 10.1145/2556288.2557361 · DOI · Paper page
- Martínez, 2014, Don’t classify ratings of affect; rank them!, IEEE Trans. Affect. Comput., 5, 314, 10.1109/TAFFC.2014.2352268 · DOI · Paper page
- Mauderer, 2014, Depth perception with gaze-contingent depth of field, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 217, 10.1145/2556288.2557089 · doi:10.1145/2556288.2557089
- Morris, 1995, Observations: sam: the self-assessment Manikinan efficient cross-cultural measurement of emotional response, J. Advert. Res., 35, 63
- Ovadia, 2004, Ratings and rankings: reconsidering the structure of values and their measurement, Int. J. Soc. Res. Method., 7, 403, 10.1080/1364557032000081654 · doi:10.1080/1364557032000081654
- Scheffe, 1952, An analysis of variance for paired comparisons, J. Am. Stat. Assoc., 47, 381, 10.2307/2281310 · DOI
- Schild, 2014, Altering gameplay behavior using stereoscopic 3d vision-based video game design, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 207, 10.1145/2556288.2557283 · doi:10.1145/2556288.2557283
- Sneddon, 2011, Cross-cultural patterns in dynamic ratings of positive and negative natural emotional behaviour, PLoS ONE, 6, e14679, 10.1371/journal.pone.0014679 · DOI
- Viswanathan, 1993, Measurement of individual differences in preference for numerical information, J. Appl. Psychol., 78, 741, 10.1037/0021-9010.78.5.741 · doi:10.1037/0021-9010.78.5.741 · Paper page
- Weigel, 2014, More than touch: understanding how people use skin as an input surface for mobile computing, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 179, 10.1145/2556288.2557239 · DOI
- Wilcoxon, 1945, Individual comparisons by ranking methods, Biometrics Bull., 80, 10.2307/3001968 · DOI · Paper page
- Yang, 2011, Ranking-based emotion recognition for music organization and retrieval, IEEE Trans. Audio Speech Lang. Process., 19, 762, 10.1109/TASL.2010.2064164 · DOI
Cited by
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205
“Zarrieß et al. (2015) used a mouse contingent reading paradigm in an evaluation study of generated text, finding that features recorded using this paradigm (e.g. reading time) provided valuable information to gauge text quality levels.”
BackgroundWith Little Power Comes Great Responsibility
Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.745 · Cited by 84
RankTrace: Relative and unbounded affect annotation
Lopes, P., Yannakakis, G. N., & Liapis, A. (2017) · 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII) · doi:10.1109/acii.2017.8273594 · Cited by 74
“We argue that these practices are detrimental to the collection and analysis of annotation signals as they factor in a multitude of subjective annotation biases that yield both constant (lack of validity) and variable (lack of reliability) deviations from the underlying ground truth [8], [9].”
BackgroundSelf-citationA Survey of Reinforcement Learning from Human Feedback
Kaufmann, T., Weng, P., Bengs, V., & Hüllermeier, E. (2023) · arXiv · doi:10.48550/arxiv.2312.14925 · Cited by 43
Metadata
- Authors
- Yannakakis, G. N., & Martínez, H. P.
- Venue
- Frontiers in ICT
- Publisher
- Frontiers Media SA
- Published
- 2015-07-30
- Volume
- 2
- ISSN
- 2297-198X (ISSN-L)
- OpenAlex
- W1527117961
- Cited by
- 78
- References
- 7
- FWCI
- 14.13
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Ratings are Overrated! https://app.cochat.ai/papers/10.3389/fict.2015.00013
- The paper
- Yannakakis, G. N., & Martínez, H. P. (2015). Ratings are Overrated! Frontiers in ICT. https://doi.org/10.3389/fict.2015.00013
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.