CoChat

Ratings are Overrated!

Open access
Yannakakis, Georgios N., Martínez, Héctor P. · 2015 · Frontiers in ICT
Cited by 78

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

Abstract

Are ratings of any use in human-computer interaction and user studies at large? If ratings are of limited use, is there a better alternative for quantitative subjective assessment? Beyond the intrinsic shortcomings of human reporting, there are a number of supplementary limitations and fundamental methodological flaws associated with rating-based questionnaires --- i.e. questionnaires that ask participants to rate their level of agreement with a given statement such as a Likert item. While the effect of these pitfalls has been largely downplayed, recent findings from diverse areas of study question the reliability of using ratings. Rank-based questionnaires --- i.e. questionnaires that ask participants to rank two or more options --- appear as the evident alternative that not only eliminates the core limitations of ratings but also simplifies the use of sound methodologies that yield more reliable models of the underlying reported construct: user emotion, preference, or opinion. This paper solicits recent findings from various disciplines interlinked with psychometrics and offers a quick guide for the use, processing and analysis of rank-based questionnaires for the unique advantages they offer. The paper challenges the traditional state-of-practice in human-computer interaction and psychometrics directly contributing towards a paradigm shift in subjective reporting.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/fict.2015.00013

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 7 references

Research works printed but not linked: 30

  1. Agresti, 1992, Analysis of ordinal paired comparison data, J. R. Stat. Soc. Ser. C Appl. Stat., 41, 287
  2. Bardram, 2013, Designing mobile health technology for bipolar disorder: a field trial of the monarca system, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 2627, 10.1145/2470654.2481364 · DOI · Paper page
  3. Baveye, 2013, A large video database for computational models of induced emotion, Proc. of Affect. Comput. Intell. Int., 13, 10.1109/ACII.2013.9 · DOI
  4. Brown, 2013, How irt can solve problems of ipsative data in forced-choice questionnaires, Psychol. Methods, 18, 36, 10.1037/a0030641 · DOI · Paper page
  5. Bryan, 2014, Isse: an interactive source separation editor, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 257, 10.1145/2556288.2557253 · DOI
  6. Chan, 1991, Response-order effects in Likert-type scales, Educ. Psychol. Meas., 51, 531, 10.1177/0013164491513002 · DOI · Paper page
  7. Chen, 2014, Duet: exploring joint interactions on a smart phone and a smart watch, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 159, 10.1145/2556288.2556955 · DOI · Paper page
  8. Čopič Pucihar, 2014, The use of surrounding visual context in handheld ar: device vs. user perspective rendering, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 197, 10.1145/2556288.2557125 · DOI
  9. Dhar, 2003, The effect of forced choice on choice, J. Market. Res., 40, 146, 10.1509/jmkr.40.2.146.19229 · DOI · Paper page
  10. Fürnkranz, 2010, Preference Learning
  11. Goyal, 2014, Effects of implicit sharing in collaborative analysis, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 129, 10.1145/2556288.2557229 · DOI
  12. Hernandez, 2014, Under pressure: sensing stress of computer users, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 51, 10.1145/2556288.2557165 · DOI · Paper page
  13. IJsselsteijn, 2008, The Game Experience Questionnaire: Development of a Self-Report Measure to Assess Player Experiences of Digital Games
  14. Jamieson, 2004, Likert scales: how to (ab) use them, Med. Educ., 38, 1217, 10.1111/j.1365-2929.2004.02012.x · DOI · Paper page
  15. Jay, 2007, Modeling the effects of delayed haptic and visual feedback in a collaborative virtual environment, ACM Trans. Comput. Hum. Interact., 14, 1275514, 10.1145/1275511 · doi:10.1145/1275511
  16. Kendall, 1938, A new measure of rank correlation, Biometrika, 81, 10.1093/biomet/30.1-2.81 · DOI · Paper page
  17. Likert, 1932, A technique for the measurement of attitudes, Arch. Psychol., 140, 1
  18. Linn, 2000, Measurement and Assessment in Teaching
  19. Mark, 2014, Stress and multitasking in everyday college life: an empirical study of online activity, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 41, 10.1145/2556288.2557361 · DOI · Paper page
  20. Martínez, 2014, Don’t classify ratings of affect; rank them!, IEEE Trans. Affect. Comput., 5, 314, 10.1109/TAFFC.2014.2352268 · DOI · Paper page
  21. Mauderer, 2014, Depth perception with gaze-contingent depth of field, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 217, 10.1145/2556288.2557089 · doi:10.1145/2556288.2557089
  22. Morris, 1995, Observations: sam: the self-assessment Manikinan efficient cross-cultural measurement of emotional response, J. Advert. Res., 35, 63
  23. Ovadia, 2004, Ratings and rankings: reconsidering the structure of values and their measurement, Int. J. Soc. Res. Method., 7, 403, 10.1080/1364557032000081654 · doi:10.1080/1364557032000081654
  24. Scheffe, 1952, An analysis of variance for paired comparisons, J. Am. Stat. Assoc., 47, 381, 10.2307/2281310 · DOI
  25. Schild, 2014, Altering gameplay behavior using stereoscopic 3d vision-based video game design, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 207, 10.1145/2556288.2557283 · doi:10.1145/2556288.2557283
  26. Sneddon, 2011, Cross-cultural patterns in dynamic ratings of positive and negative natural emotional behaviour, PLoS ONE, 6, e14679, 10.1371/journal.pone.0014679 · DOI
  27. Viswanathan, 1993, Measurement of individual differences in preference for numerical information, J. Appl. Psychol., 78, 741, 10.1037/0021-9010.78.5.741 · doi:10.1037/0021-9010.78.5.741 · Paper page
  28. Weigel, 2014, More than touch: understanding how people use skin as an input surface for mobile computing, Proc. SIGCHI Conf. Hum. Factor. Comput. Syst., 179, 10.1145/2556288.2557239 · DOI
  29. Wilcoxon, 1945, Individual comparisons by ranking methods, Biometrics Bull., 80, 10.2307/3001968 · DOI · Paper page
  30. Yang, 2011, Ranking-based emotion recognition for music organization and retrieval, IEEE Trans. Audio Speech Lang. Process., 19, 762, 10.1109/TASL.2010.2064164 · DOI

Cited by

See all 78

Metadata

Authors
Yannakakis, G. N., & Martínez, H. P.
Venue
Frontiers in ICT
Published
2015-07-30
Volume
2
ISSN
2297-198X (ISSN-L)
OpenAlex
W1527117961
Cited by
78
References
7
FWCI
14.13
Type
journal article
Language
en
Open access
diamond

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Ratings are Overrated! https://app.cochat.ai/papers/10.3389/fict.2015.00013
The paper
Yannakakis, G. N., & Martínez, H. P. (2015). Ratings are Overrated! Frontiers in ICT. https://doi.org/10.3389/fict.2015.00013

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.