Ranking vs. Preference: A Comparative Study of Self-reporting
Open accessThe DOI matches its Crossref record; 18 of the 19 references it cites were found in a registry or an index.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Scalar feedback is more susceptible to inconsistency between annotators and suffers from bias due to the order in which examples are presented (Yannakakis and Hallam, 2011).”
From Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023) · cites this paper as Yannakakis and Hallam, 2011
In the citing paper
“Thus, quality assessment rests on human judgment and varies significantly based on the individual evaluator's perspective [45,208,222].”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [208]
In the citing paper
“Nonetheless, in order to generate high-quality text, it is crucial to note that the task of distinguishing between high and low-quality text is inherently subjective and challenging, with end users being humans.”
From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [208]
Citations
Cites
Coefficient Alpha and the Internal Structure of Tests
Cronbach, L. J. (1951) · Psychometrika · doi:10.1007/bf02310555 · Cited by 43,931
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398
Journal of The American Statistical Association
Fienberg, S. E. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,125
Journal of the American Statistical Association
Koehler, A. B. (1995) · International Journal of Forecasting · doi:10.1016/0169-2070(95)90067-5 · Cited by 5,337
On the Theory of Scales of Measurement
Stevens, S. S. (1946) · Science · doi:10.1126/science.103.2684.677 · Cited by 4,761 · cited 2×
“The vast majority of psychometric and user studies have adopted a type of rating report to capture the subjective assessment of the experiment participants ([11] among others).”
BackgroundMeasurement and Assessment in Teaching
Linn, R. L., & Gronlund, N. E. (1994) · Medical Entomology and Zoology · Cited by 1,276
Journal of the Royal Statistical Society: Series C (Applied Statistics)
(2023) · Journal of the Royal Statistical Society Series C (Applied Statistics) · doi:10.1111/(issn)1467-9876 · Cited by 1,178
An Analysis of Variance for Paired Comparisons
Scheffé, H. (1952) · Journal of the American Statistical Association · doi:10.1080/01621459.1952.10501179 · Cited by 369
Playware technology for physically activating play
Lund, H. H., Klitbo, T., & Jessen, C. (2005) · Artificial Life and Robotics · doi:10.1007/s10015-005-0350-z · Cited by 146
Towards affective camera control in games
Yannakakis, G. N., Martínez, H. P., & Jhala, A. H. (2010) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-010-9078-0 · Cited by 128
Self-citation
Research works printed but not linked: 3
- Likert, R.: A technique for the measurement of attitudes. Archives of Psychology 140, 1–55 (1932)
- Poels, K., IJsselsteijn, W.: Development and validation of the game experience questionnaire. In: FUGA Workshop Mini-Symposium, Helsinki, Finland (2008)
- Read, J., MacFarlane, S., Cassey, C.: Endurability, engagement and expectations. In: Proceedings of International Conference for Interaction Design and Children (2002)
Cited by
Stress and anxiety detection using facial cues from videos
Giannakakis, G., Pediaditis, M., Manousos, D., Kazantzaki, E., Chiarugi, F., Simos, P. G., Marias, K., & Tsiknakis, M. (2017) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2016.06.020 · Cited by 342
LIRIS-ACCEDE: A Video Database for Affective Content Analysis
Baveye, Y., Dellandrea, E., Chamaret, C., & Chen, L. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2396531 · Cited by 306
Learning deep physiological models of affect
Martinez, H. P., Bengio, Y., & Yannakakis, G. N. (2013) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2013.2247823 · Cited by 287
“Independently of model input, the use of preference learning models—which are trained and evaluated using within-participant differences—automatically minimizes the effects of between-participants physiological differences (as noted in [33], [12] among other studies).”
BackgroundMethodInfluentialSelf-citationBest practices for the human evaluation of automatically generated text
van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019) · Proceedings of the 12th International Conference on Natural Language Generation · doi:10.18653/v1/w19-8643 · Cited by 205
Yannakakis, G. N. (2012) · Proceedings of the 9th conference on Computing Frontiers · doi:10.1145/2212908.2212954 · Cited by 179
Self-citation
Metadata
- Authors
- Yannakakis, G. N., & Hallam, J.
- Affiliations
- IT University of Copenhagen (DK)
- University of Southern Denmark (DK)
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer Berlin Heidelberg
- Published
- 2011
- Pages
- 437-446
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W133537955
- Cited by
- 90
- References
- 19
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 5 · 2023: 3 · 2022: 4 · 2021: 2 · 2020: 6 · 2019: 7 · 2018: 5 · 2017: 11 · 2016: 7 · 2015: 11 · 2014: 12 · 2013: 7 · 2012: 4
- FWCI
- 12.20
- Type
- conference paper
- Language
- en
- Open access
- green
- www.um.edu.mt
- Licence
- Locations
- Lecture notes in computer science (publishedVersion)
- OAR@UM (University of Malta) (acceptedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Psychology › Developmental and Educational Psychology › Educational Games and Gamification
- Keywords
- self-report, rating scales, pairwise comparison, user survey, order effects, questionnaire design, user experience evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Ranking vs. Preference: A Comparative Study of Self-reporting. https://app.cochat.ai/papers/10.1007/978-3-642-24600-5_47
- The paper
- Yannakakis, G. N., & Hallam, J. (2011). Ranking vs. Preference: A Comparative Study of Self-reporting. Lecture Notes in Computer Science. https://doi.org/10.1007/978-3-642-24600-5_47
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.