Don’t Classify Ratings of Affect; Rank Them!
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
How should affect be appropriately annotated and how should machine learning best be employed to map manifestations of affect to affect annotations? What is the use of ratings of affect for the study of affective computing and how should we treat them? These are the key questions this paper attempts to address by investigating the impact of dissimilar representations of annotated affect on the efficacy of affect modelling. In particular, we compare several different binary-class and pairwise preference representations for automatically learning from ratings of affect. The representations are compared and tested on three datasets: one synthetic dataset (testing “in vitro ”) and two affective datasets (testing “in vivo”). The synthetic dataset couples a number of attributes with generated rating values. The two affective datasets contain physiological and contextual user attributes, and speech attributes, respectively; these attributes are coupled with ratings of various affective and cognitive states. The main results of the paper suggest that ratings (when used) should be naturally transformed to ordinal (ranked) representations for obtaining more reliable and generalisable models of affect. The findings of this paper have a direct impact on affect annotation and modelling research but, most importantly, challenge the traditional state-of-practice in affective computing and psychometrics at large.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
On the Theory of Scales of Measurement
Stevens, S. S. (1946) · Science · doi:10.1126/science.103.2684.677 · Cited by 4,761 · cited 3×
“Such effects, however, are not trivial to show through a data modelling approach, since the objec-tive ground truth is fundamentally ill-defined when ratings are treated as numerical values [5].”
BackgroundWhat are emotions? And how can they be measured?
Scherer, K. R. (2005) · Social Science Information · doi:10.1177/0539018405058216 · Cited by 4,264 · cited 1×
“Index Terms—affect annotation, affect modelling, ratings, ranks, preference learning, classification, computer games, Sensitive Artificial Listener (SAL) corpus F”
BackgroundAffect Detection: An Interdisciplinary Review of Models, Methods, and Their Applications
Calvo, R. A., & D'Mello, S. (2010) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.1 · Cited by 1,893
The visual analogue scale: Its use in pain measurement
Langley, G. B., & Sheppeard, H. (1985) · Rheumatology International · doi:10.1007/bf00541514 · Cited by 619 · cited 3×
“The three machine learning alternatives for learning from rating-based affect annotations, namely regression , classification and preference learning , are discussed in detail in this section.”
BackgroundNicolaou, M. A., Gunes, H., & Pantic, M. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.9 · Cited by 450 · cited 1×
“The three machine learning alternatives for learning from rating-based affect annotations, namely regression , classification and preference learning , are discussed in detail in this section.”
Background(2013) · doi:10.4324/9780203763247 · Cited by 400
Learning deep physiological models of affect
Martinez, H. P., Bengio, Y., & Yannakakis, G. N. (2013) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2013.2247823 · Cited by 287 · cited 1×
“have already demonstrated their efficacy in affect modelling tasks [28], [29], [30], [40].”
BackgroundSelf-citationCall Center Stress Recognition with Person-Specific Models
Hernandez, J., Morris, R. R., & Picard, R. W. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24600-5_16 · Cited by 210 · cited 3×
“…is typically minimised by gathering several reports from the same user and normalising the responses with the minimum and maximum values used (see [15] among others) which, in turn, exacerbates the non-linear scale bias and adds other types of experimental bias as the scale is artificially…”
BackgroundRanking vs. Preference: A Comparative Study of Self-reporting
Yannakakis, G. N., & Hallam, J. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-24600-5_47 · Cited by 90
Self-citation
Research works printed but not linked: 35
- kolmogorov, 1957, Dokl Akad Nauk SSSR
- joachims, 0, Text categorization with support vector machines: Learning with many relevant features, Proc 10th Eur Conf Mach Learn, 137
- 10.1007/978-3-642-24600-5_55 · doi:10.1007/978-3-642-24600-5_55
- 10.1016/j.ijhcs.2008.06.004 · DOI
- 10.1007/s11257-010-9078-0 · DOI
- 10.2307/2332226 · DOI
- 10.1145/1007730.1007733 · DOI · Paper page
- crammer, 2001, Pranking with ranking, Adv Neural Inf Process Syst, 14, 641
- 10.1145/1102351.1102369 · DOI · Paper page
- 10.1016/j.specom.2007.01.010 · DOI · Paper page
- 10.1109/ICME.2005.1521579 · DOI · Paper page
- 10.1109/ICME.2010.5583101 · DOI
- 10.1080/1364557032000081654 · doi:10.1080/1364557032000081654
- 10.1109/FG.2013.6553804 · DOI
- 10.1109/TASL.2010.2064164 · DOI
- 10.1371/journal.pone.0014679 · DOI
- 10.1007/s10994-008-5097-z · DOI
- morris, 1995, Observations: Sam: The self-assessment manikinan efficient cross-cultural measurement of emotional response, J Advert Res, 35, 63
- fürnkranz, 2010, Preference Learning
- 10.1007/978-3-642-24600-5_30 · DOI
- 10.1007/978-3-642-24600-5_60 · DOI
- 10.1007/978-3-642-24600-5_23 · DOI
- likert, 1932, A technique for the measurement of attitudes, Arch Psychol, 22, 55
- healey, 0, Recording affect in the field: Towards methods and metrics for improving ground truth labels, Proc 4th Int Conf Affective Comput Intell Interaction, 6974, 107, 10.1007/978-3-642-24600-5_14 · DOI
- mitchell, 1997, Machine Learning
- 10.1109/T-AFFC.2012.15 · DOI
- 10.1145/1180995.1181029 · DOI · Paper page
- meng, 0, Multi-score learning for affect recognition: The case of body postures, Proc 4th Int Conf Affective Comput Intell Interaction, 6974, 225, 10.1007/978-3-642-24600-5_26 · DOI
- agresti, 2010, Analysis of Ordinal Categorical Data, 656, 10.1002/9780470594001 · DOI · Paper page
- baker, 0, The dynamics between student affect and behavior occurring outside of educational software, Proc 4th Int Conf Affective Comput Intell Interaction, 6974, 14, 10.1007/978-3-642-24600-5_5 · DOI
- georgiou, 0, ‘That’s aggravating, very aggravating’: Is it possible to classify behaviors in couple interactions using automatically derived lexical features?, Proc 4th Int Conf Affective Comput Intell Interaction, 87
- 10.1145/1877826.1877832 · DOI
- 10.1109/TPAMI.2009.107 · DOI
- 10.1007/s12193-013-0122-3 · DOI
- 10.1007/978-3-642-24600-5_19 · DOI
Cited by
Gatt, A., & Krahmer, E. (2018) · Journal of Artificial Intelligence Research · doi:10.1613/jair.5477 · Cited by 776
“However, some recent work in affective computing has questioned the use of ratings, comparing them to ranking-based and physiological methods (e.g. Martinez et al., 2014; Yannakakis & Mart́ınez, 2015).”
BackgroundA Review of Human Activity Recognition Methods
Vrigkas, M., Nikou, C., & Kakadiaris, I. A. (2015) · Frontiers in Robotics and AI · doi:10.3389/frobt.2015.00028 · Cited by 566
Lotfian, R., & Busso, C. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2736999 · Cited by 329
Roldán, J. J., Crespo, E., Martín-Barrio, A., Peña-Tapia, E., & Barrientos, A. (2019) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2019.05.004 · Cited by 211
Best practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205
“Recent studies have found that participant ratings are more reliable, consistent, and are less prone to order effects when they involve ranking rather than Likert scales (Martinez et al., 2014; Yannakakis and Mart´ınez, 2015; Yannakakis and Hallam, 2011).”
Background
Metadata
- Authors
- Martinez, H. P., Yannakakis, G. N., & Hallam, J.
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2014-07-01
- Volume
- 5
- Issue
- 3
- Pages
- 314-326
- ISSN
- 1949-3045 (ISSN-L)
- OpenAlex
- W1993008008
- Cited by
- 109
- References
- 9
- FWCI
- 15.08
- Type
- journal article
- Language
- en
- Open access
- green
- Funders
- EU-funded FP7 ICT ILearnRW (318803)
- Experiments reported in this paper run at the computational facilities procured through the European Regional Development Fund (ERDF-080)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Don’t Classify Ratings of Affect; Rank Them! https://app.cochat.ai/papers/10.1109/taffc.2014.2352268
- The paper
- Martinez, H. P., Yannakakis, G. N., & Hallam, J. (2014). Don’t Classify Ratings of Affect; Rank Them! IEEE Transactions on Affective Computing. https://doi.org/10.1109/taffc.2014.2352268
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.