The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
Best Practices for QoE Crowdtesting: QoE Assessment With Crowdsourcing
Abstract
Quality of Experience (QoE) in multimedia applications is closely linked to the end users' perception and therefore its assessment requires subjective user studies in order to evaluate the degree of delight or annoyance as experienced by the users. QoE crowdtesting refers to QoE assessment using crowdsourcing, where anonymous test subjects conduct subjective tests remotely in their preferred environment. The advantages of QoE crowdtesting lie not only in the reduced time and costs for the tests, but also in a large and diverse panel of international, geographically distributed users in realistic user settings. However, conceptual and technical challenges emerge due to the remote test settings. Key issues arising from QoE crowdtesting include the reliability of user ratings, the influence of incentives, payment schemes and the unknown environmental context of the tests on the results. In order to counter these issues, strategies and methods need to be developed, included in the test design, and also implemented in the actual test campaign, while statistical methods are required to identify reliable user ratings and to ensure high data quality. This contribution therefore provides a collection of best practices addressing these issues based on our experience gained in a large set of conducted QoE crowdtesting studies. The focus of this article is in particular on the issue of reliability and we use video quality assessment as an example for the proposed best practices, showing that our recommended two-stage QoE crowdtesting design leads to more reliable results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Answering the Call for a Standard Reliability Measure for Coding Data
Hayes, A. F., & Krippendorff, K. (2007) · Communication Methods and Measures · doi:10.1080/19312450709336664 · Cited by 4,225 · cited 1×
“Krippendorff’s α [63] is a reliability coefficient to measure the agreement among test subjects.”
BackgroundLabeling images with a computer game
von Ahn, L., & Dabbish, L. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985733 · Cited by 2,204
Crowdsourcing user studies with Mechanical Turk
Kittur, A., Chi, E. H., & Suh, B. (2008) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1357054.1357127 · Cited by 1,923
Towards an integrated crowdsourcing definition
Estellés-Arolas, E., & González-Ladrón-de-Guevara, F. (2012) · Journal of Information Science · doi:10.1177/0165551512437638 · Cited by 1,858
Quality management on Amazon Mechanical Turk
Ipeirotis, P. G., Provost, F., & Wang, J. (2010) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1837885.1837906 · Cited by 973
von Ahn, L. (2006) · Computer · doi:10.1109/mc.2006.196 · Cited by 955
Harnessing the wisdom of crowds in wikipedia: quality through coordination
Kittur, A., & Kraut, R. E. (2008) · Proceedings of the 2008 ACM conference on Computer supported cooperative work · doi:10.1145/1460563.1460572 · Cited by 680
Are your participants gaming the system?: screening mechanical turk workers
Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753688 · Cited by 418
Quantification of YouTube QoE via Crowdsourcing
Hoßfeld, T., Seufert, M., Hirth, M., Zinner, T., Tran-Gia, P., & Schatz, R. (2011) · 2011 IEEE International Symposium on Multimedia · doi:10.1109/ism.2011.87 · Cited by 338
Self-citationCrowdsourcing for relevance evaluation
Alonso, O., Rose, D. E., & Stewart, B. (2008) · ACM SIGIR Forum · doi:10.1145/1480506.1480508 · Cited by 329
Data quality from crowdsourcing: a study of annotation selection criteria
Hsueh, P.-Y., Melville, P., & Sindhwani, V. (2009) · Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing - HLT '09 · doi:10.3115/1564131.1564137 · Cited by 251
Initial delay vs. interruptions: Between the devil and the deep blue sea
Hossfeld, T., Egger, S., Schatz, R., Fiedler, M., Masuch, K., & Lorentzen, C. (2012) · 2012 Fourth International Workshop on Quality of Multimedia Experience · doi:10.1109/qomex.2012.6263849 · Cited by 245
Self-citationTurKit: tools for iterative tasks on mechanical Turk
Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. (2009) · Proceedings of the ACM SIGKDD Workshop on Human Computation · doi:10.1145/1600150.1600159 · Cited by 237
Designing incentives for inexpert human raters
Shaw, A. D., Horton, J. J., & Chen, D. L. (2011) · Proceedings of the ACM 2011 conference on Computer supported cooperative work · doi:10.1145/1958824.1958865 · Cited by 236
Factors influencing quality of experience of commonly used mobile applications
Ickin, S., Wac, K., Fiedler, M., Janowski, L., Hong, J.-H., & Dey, A. K. (2012) · IEEE Communications Magazine · doi:10.1109/mcom.2012.6178833 · Cited by 225
CROWDMOS: An approach for crowdsourcing mean opinion score studies
Ribeiro, F., Florencio, D., Zhang, C., & Seltzer, M. (2011) · 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2011.5946971 · Cited by 221
Challenges of QoE management for cloud applications
Hobfeld, T., Schatz, R., Varela, M., & Timmerer, C. (2012) · IEEE Communications Magazine · doi:10.1109/mcom.2012.6178831 · Cited by 212
Programmatic gold: targeted and scalable quality assurance in crowdsourcing
Oleson, D., Sorokin, A. A., Laughlin, G., Hester, V., Le, J., & Biewald, L. (2011) · Cited by 195
A crowdsourceable QoE evaluation framework for multimedia content
Chen, K.-T., Wu, C.-C., Chang, Y.-C., & Lei, C.-L. (2009) · Proceedings of the 17th ACM international conference on Multimedia · doi:10.1145/1631272.1631339 · Cited by 192
Quality through flow and immersion: gamifying crowdsourced relevance assessments
Eickhoff, C., Harris, C. G., de Vries, A. P., & Srinivasan, P. (2012) · Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/2348283.2348400 · Cited by 190
Le, J., Edmonds, A., Hester, V., & Biewald, L. (2010) · Cited by 182
Hossfeld, T., Schatz, R., & Egger, S. (2011) · 2011 Third International Workshop on Quality of Multimedia Experience · doi:10.1109/qomex.2011.6065690 · Cited by 168
Self-citationIncreasing cheat robustness of crowdsourcing tasks
Eickhoff, C., & de Vries, A. P. (2013) · Information Retrieval · doi:10.1007/s10791-011-9181-9 · Cited by 167
Subjective assessment of H.264/AVC video sequences transmitted over a noisy channel
Francesca De Simone, A., Naccari, M., Tagliasacchi, M., Frederic Dufaux, A., Tubaro, S., & Ebrahimi, T. (2009) · International Conference on Quality of Multimedia Experience (QoMEX) · doi:10.1109/qomex.2009.5246952 · Cited by 142
Anatomy of a Crowdsourcing Platform - Using the Example of Microworkers.com
Hirth, M., Hoßfeld, T., & Tran‐Gia, P. (2011) · International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing (IMIS) · doi:10.1109/imis.2011.89 · Cited by 141
Self-citationDecision-Theoretic Control of Crowd-Sourced Workflows
Dai, P., Mausam, M., & Weld, D. S. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7760 · Cited by 126
A H.264/AVC video database for the evaluation of quality metrics
Francesca De Simone, A., Tagliasacchi, M., Naccari, M., Tubaro, S., & Ebrahimi, T. (2010) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2010.5496296 · Cited by 126
Quadrant of euphoria: a crowdsourcing platform for QoE assessment
Chen, K.-T., Chang, C.-J., Wu, C., et al. (2010) · IEEE Network · doi:10.1109/mnet.2010.5430141 · Cited by 112
The memory effect and its implications on Web QoE modeling
Hosfeld, T., Biedermann, S., Schatz, R., Platzer, A., Egger, S., & Fiedler, M. (2011) · International Teletraffic Congress · doi:10.5555/2043468.2043485 · Cited by 104
QualityCrowd — A framework for crowd-based quality evaluation
Keimel, C., Habigt, J., Horch, C., & Diepold, K. J. (2012) · Picture Coding Symposium · doi:10.1109/pcs.2012.6213338 · Cited by 94
Self-citationCrowdsourcing research opportunities
Sabou, M., Bontcheva, K., & Scharl, A. (2012) · International Conference on Knowledge Management and Knowledge Technologies · doi:10.1145/2362456.2362479 · Cited by 93
Crowdsourcing Multimedia QoE Evaluation: A Trusted Framework
Wu, C., Chen, K.-T., Chang, Y.-C., & Lei, C. (2013) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2013.2241043 · Cited by 92
HodgeRank on Random Graphs for Subjective Video Quality Assessment
Xu, Q., Huang, Q., Jiang, T., et al. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2012.2190924 · Cited by 88
Crowdsourcing for Affective Annotation of Video: Development of a Viewer-reported Boredom Corpus
Soleymani, M., & Larson, M. (2010) · Archive ouverte UNIGE (University of Geneva) · Cited by 83
You're Hired! An Examination of Crowdsourcing Incentive Models in Human Resource Tasks
Harris, C. G. (2011) · Cited by 77
Honesty in an Online Labor Market
Suri, S., Goldstein, D. G., & Mason, W. (2013) · Cited by 69
How Crowdsourcable is Your Task
Eickhoff, C., & de Vries, A. P. (2011) · Data Archiving and Networked Services (DANS) · Cited by 66
Shepherding the crowd: managing and providing feedback to crowd workers
Dow, S. P., Kulkarni, A. P., Bunge, B., Nguyen, T., Klemmer, S., & Hartmann, B. (2011) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/1979742.1979826 · Cited by 59
Quality assessment of multidimensional video scalability
Lee, J., Francesca, D. S., Ebrahimi, T., Ramzan, N., & Ebroul Izquierdo, M. (2012) · IEEE Communications Magazine · doi:10.1109/mcom.2012.6178832 · Cited by 43
The Impact of Test Duration on User Fatigue and Reliability of Subjective Quality Ratings
Schatz, R., Egger, S., & Masuch, K. (2012) · Journal of the Audio Engineering Society · Cited by 37
Impact of screening technique on crowdsourcing QoE assessments
Gardlo, B., Ries, M., & Hoßfeld, T. (2012) · International Conference Radioelektronika · Cited by 24
Self-citation
Cited by
Waterloo Exploration Database: New Challenges for Image Quality Assessment Models
Ma, K., Duanmu, Z., Wu, Q., Wang, Z., Yong, H., Li, H., & Zhang, L. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2016.2631888 · Cited by 789
Massive Online Crowdsourced Study of Subjective and Objective Picture Quality
Ghadiyaram, D., & Bovik, A. C. (2016) · IEEE Transactions on Image Processing · doi:10.1109/tip.2015.2500021 · Cited by 782
A survey of the use of crowdsourcing in software engineering
Mao, K., Capra, L., Harman, M., & Jia, Y. (2017) · Journal of Systems and Software · doi:10.1016/j.jss.2016.09.015 · Cited by 385
Two-Level Approach for No-Reference Consumer Video Quality Assessment
Korhonen, J. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2923051 · Cited by 354
Large-Scale Study of Perceptual Video Quality
Sinno, Z., & Bovik, A. C. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2869673 · Cited by 310
Metadata
- Authors
- Hossfeld, T., Keimel, C., Hirth, M., Gardlo, B., Habigt, J., Diepold, K., & Tran-Gia, P.
- Venue
- IEEE Transactions on Multimedia
- Published
- 2014-02
- Volume
- 16
- Issue
- 2
- Pages
- 541-558
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2163031058
- Cited by
- 283
- References
- 41
- FWCI
- 35.01
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.