The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Input-agreement
Abstract
Since its introduction at CHI 2004, the ESP Game has inspired many similar games that share the goal of gathering data from players. This paper introduces a new mechanism for collecting labeled data using "games with a purpose." In this mechanism, players are provided with either the same or a different object, and asked to describe that object to each other. Based on each other's descriptions, players must decide whether they have the same object or not. We explain why this new mechanism is superior for input data with certain characteristics, introduce an enjoyable new game called "TagATune" that collects tags for music clips via this mechanism, and present findings on the data that is collected by this game.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Labeling images with a computer game
von Ahn, L., & Dabbish, L. (2004) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985733 · Cited by 2,234
Designing games with a purpose
von Ahn, L., & Dabbish, L. (2008) · Communications of the ACM · doi:10.1145/1378704.1378719 · Cited by 1,186
von Ahn, L. (2006) · Computer · doi:10.1109/mc.2006.196 · Cited by 968
Social Tagging and Music Information Retrieval
Lamere, P. (2008) · Journal of New Music Research · doi:10.1080/09298210802479284 · Cited by 280
Games with a Purpose for the Semantic Web
Siorpaes, K., & Hepp, M. F. (2008) · IEEE Intelligent Systems · doi:10.1109/mis.2008.45 · Cited by 215
Tagatune: A Game For Music And Sound Annotation.
Law, E., von Ahn, L., Dannenberg, R. B., & Crawford, M. (2007) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1415567 · Cited by 195
Self-citationHacker, S., & von Ahn, L. (2009) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1518882 · Cited by 142
A Web-Based Game for Collecting Music Metadata
Mandel, M., & Ellis, D. P. W. (2008) · Journal of New Music Research · doi:10.1080/09298210802479300 · Cited by 129
Moodswings: A Collaborative Game For Music Mood Label Collection.
Kim, Y. E., Schmidt, E. M., & Emelle, L. (2008) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1416585 · Cited by 106
A Game-Based Approach For Collecting Semantic Annotations Of Music.
Turnbull, D., Liu, R., Barrington, L., & Lanckriet, G. R. G. (2007) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1416463 · Cited by 98
Cited by
Sorted by citations
Items 201–258 of 258
Input-evaluation: A new mechanism for collecting data using games with a purpose
Gencer, A. E., Güngör, T., Gürer, A., & Özsoy, A. S. (2012) · IEEE Symposium on Computers and Communications (ISCC) · doi:10.1109/iscc.2012.6249301 · Cited by 1
Combino - A GWAP for Generating Combined Tags
Störkle, F. (2012) · Cited by 1
Play for Me: Image Segmentation via Seamless Playsourcing
Hennersperger, C., & Baust, M. (2016) · The Computer Games Journal · doi:10.1007/s40869-016-0030-3 · Cited by 1
GWAP Design for a Mobile Geo-Tagging System with Confident Verification
Tsai, T.-C., & Yang, T.-R. (2011) · IEEE International Conference on Computational Science and Engineering (CSE) · doi:10.1109/cse.2011.61 · Cited by 1
Collective Action and Human Computation
Novak, J. (2013) · Springer eBooks · doi:10.1007/978-1-4614-8806-4_33 · Cited by 1
Listening Level Changes Music Similarity.
Terrell, M. J., Fazekas, G., Simpson, A., Smith, J. B. L., & Dixon, S. (2012) · doi:10.5281/zenodo.1415709 · Cited by 1
Ensemble: A Hybrid Human-Machine System For Generating Melody Scores From Audio.
Tse, T., Salamon, J., Williams, A. C., Jiang, H., & Law, E. (2016) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1416707 · Cited by 1
Self-citationTopical Video Search: Analysing Video Concept Annotation through Crowdsourcing Games
Gligorov, R., Hildebrand, M., van Ossenbruggen, J., Aroyo, L. M., & Guus Schreiber, A. (2017) · Human Computation · doi:10.15346/hc.v4i1.77 · Cited by 1
Putting User Reputation on the Map
Barz, B., van Dijk, T. C., Spaan, B., & Denzler, J. (2018) · doi:10.1145/3282933.3282937 · Cited by 1
Spotivibes: Tagging playlist vibes with colors
Abderrazik, H., Angela, G., Brouwer, H., Janse, H., Lutz, S., Smitskamp, G., Manolios, S., Liem, C. C. S., Brusilovsky, P., de Gemmis, M., et al. (2019) · Research Repository (Delft University of Technology) · Cited by 1
Functional Data Analysis: An Interdisciplinary Statistical Topic
Delsol, L., Ferraty, F., & Calvo, A. M. (2011) · Chapman & Hall/CRC eBooks · doi:10.1201/b11429-22 · Cited by 1
Wei, Z., Paliyawan, P., & Thawonmas, R. (2022) · IEEE Access · doi:10.1109/access.2022.3169272 · Cited by 1
The Influence of Crowdsourcing Business Model into Artificial Intelligence
Szopa, A. (2019) · IGI Global eBooks · doi:10.4018/978-1-5225-8362-2.ch013 · Cited by 1
Characterizing Swiss Alpine Lakes: from Wikipedia to Citizen Science
Lin, Y., & Gática-Pérez, D. (2023) · ACM Journal on Computing and Sustainable Societies · doi:10.1145/3617128 · Cited by 1
Chee, H. E. M., Wang, J., Guo, Z., Ma, W., & Zhang, M. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3758310 · Cited by 1
mTagATune: Mobile TagATune -- Audio Files Tagging Mobile Game
Díaz, F. J., Queiruga, C. A., Ferraresso, A., & Larghi, J. L. (2011) · International Conference on Mobile Business · doi:10.1109/icmb.2011.39 · Cited by 1
Spot the Odd Song Out: Similarity Model Adaptation and Analysis using Relative Human Ratings
Wolff, D. (2014) · City Research Online (City University London) · Cited by 1
Bertocco, M. (2011) · Cited by 1
Andro, M. (2016) · theses.fr (ABES) · doi:10.70675/c8e66de5z53fbz4bd9zb0dcz41accc4a6759 · Cited by 1
Retrieval and Annotation of Music Using Latent Semantic Models
Lévy, M. (2012) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 1
Inskip, C. (2010) · City Research Online (City University London)
Exploiting structure in man-made environments
Aydemir, A. (2012) · KTH Publication Database DiVA (KTH Royal Institute of Technology)
Perception of Parameter Variations in Linear Fractal Images
Hepting, D. H., & Latifi, L. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15470-6_22
Game with a Purpose for Tagging Emotions in Dialect Arabic Speech
Rizkallah, M., Abdulaal, O., Bastawisy, M. E., Hamed, I., & Abdennadher, S. (2014) · International Conference on Signal-Image Technology & Internet-Based Systems (SITIS) · doi:10.1109/sitis.2014.109
An Implementation of a Local Issue Sharing System Based on Incentive Mechanism
Tsuboi, T., Hayakawa, T., Suzuki, R., & Itō, T. (2014) · IEEE International Conference on Service-Oriented Computing and Applications (SOCA) · doi:10.1109/soca.2014.35
Performance and perceptions of human computation games for image tagging
Goh, D. H., Lee, C. S., & Chua, A. Y. K. (2011) · ACM Symposium on Applied Computing · doi:10.1145/1982185.1982433
Detecting user types in object ranking decisions
Lu, X., Schaal, M., Adalı, S., & Raju, A. K. (2009) · International Conference on Management of Emergent Digital EcoSystems (MEDES) · doi:10.1145/1643823.1643851
ANOTAÇÃO COLABORATIVA DE CONTEÚDOS UTILIZANDO JOGOS COM PROPÓSITO
Pinto, J. P. L. G. (2012) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT)
A Game with a Purpose for Recommender Systems
Smyth, B., Rafter, R., & Banks, S. (2015) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v3i1.13241
Content-aware visualizations of audio data in diverse contexts
Ness, S. R. (2009) · UVic’s Research and Learning Repository (University of Victoria)
Pe‐Than, E. P. P. (2016) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/66006
Learning content-based metrics for music similarity
Dieleman, S., & Schrauwen, B. (2012) · Ghent University Academic Bibliography (Ghent University)
Duong, T. (2014) · Infoscience (Ecole Polytechnique Fédérale de Lausanne)
A Novel Application of Gamification for Collecting High-Level Design Information
Cheong, H., Li, W., & Iorio, F. (2015) · ASME International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (IDETC/CIE) · doi:10.1115/detc2015-47383
Exploring Human Computation and Social Computing to Inform the Design Process
Romani, R., & Baranauskas, M. C. C. (2013) · International Conference on Enterprise Information Systems · doi:10.5220/0004434300670074
Serious Game Framework for Design of Medical Applications
Xu, Y., Ramanathan, J., & Ramnath, R. (2012) · IGI Global eBooks · doi:10.4018/978-1-4666-0149-9.ch018
Feedback and Timing in a Crowdsourcing GamePreprint
Freedman, G., Punjasthitkul, S., Seidman, M., & Flanagan, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.02182
Gamification for informal terms lexicon building
Alvarado, F. H. C., & Chen, Y.-S. (2016) · Advances in Social Networks Analysis and Mining · doi:10.5555/3192424.3192678
The Influence of Crowdsourcing Business Model into Artificial Intelligence
Szopa, A. (2016) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-5225-1656-9.ch002
Audio-based Distributional Semantic Models for Music Auto-tagging and Similarity MeasurementPreprint
Karamanolakis, G., Iosif, E., Zlatintsi, A., Pikrakis, A., & Potamianos, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08391
Fusion multi-niveaux par boosting pour le tagging automatique
Gaël Richard, A., Essid, S., & Lagrange, M. (2013)
The Augmented Shopping Experience with Intuitive Intelligence and Machine Learning
Kreisel, A. (2019) · Maryland Shared Open Access Repository (USMAI Consortium) · doi:10.13016/m2fil9-fgng
Human based Computation: A solution to Computer Vision Problems
Kaur, P., & Kumar, R. (2018) · International Conference on Advances in Computing, Communication Control and Networking (ICAC3N) · doi:10.1109/icacccn.2018.8748543
Supporting Effective Collective Ideation at Scale
Siangliulue, K. (2017) · Digital Access to Scholarship at Harvard (DASH) (Harvard University)
Gamification for informal terms lexicon building
Alvarado, F. H. C., & Chen, Y.-S. (2016) · IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1109/asonam.2016.7752418
The Influence of Crowdsourcing Business Model Into Artificial Intelligence
Szopa, A. (2019) · IGI Global eBooks · doi:10.4018/978-1-5225-8182-6.ch027
The Use of Games in Citizen Science Based on Findings from the EyeWire User Study
Jaśkowiec, M., & Kowalska, M. (2023) · Games and Culture · doi:10.1177/15554120231196260
A Quest Through Interconnected Datasets: Lessons From Highly-Cited ICASSP Papers
Liem, C. C. S., Taşcılar, D., & Demetriou, A. M. (2024) · International Workshop on Content-Based Multimedia Indexing (CBMI) · doi:10.1109/cbmi62980.2024.10859219
Digital Games to Build a Knowledge Base
Goria, S. (2025) · Wiley eBooks · doi:10.1002/9781394372591.ch12
Algorithmes de recommandation musicale
Maillet, F. (2009) · doi:10.71781/10290
Bellec, G., Elowsson, A., Friberg, A., Wolff, D., & Weyde, T. (2013) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.850197
van Zoest, M. (2018) · Research Repository (Delft University of Technology)
Fusion multi-niveaux par boosting pour le tagging automatique
Foucard, R. (2013) · theses.fr (ABES) · doi:10.70675/89c56b4bz28aaz483aza287zd3acfefceba2
Wang, Z., Ding, Y., Gao, M., Ren, Z., Mariakakis, A. T., Shi, Y., & Wang, Y. (2026) · Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies · doi:10.1145/3831976
Law, E., & Yin, M. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.008
Self-citationDoing Human-Centered Machine Learning
Fiebrink, R., Gillies, M. F. P., & Ramos, G. (2026) · Cambridge University Press eBooks · doi:10.1017/9781108872225.006
Truth discovery under resource constraints
Etuk, A. (2015)
Metadata
- Authors
- Law, E., & von Ahn, L.
- Affiliations
- Carnegie Mellon University (US)
- Carnegie Mellon University, Pittsburgh, PA, USA
- ORCID
- Law, Edith
- Venue
- Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2009-04-04
- Pages
- 1197-1206
- OpenAlex
- W2067329295
- Semantic Scholar
- 72dd231084d2fee5c31343b395ae372b68389ba9
- Cited by
- 248
- References
- 12
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 3 · 2023: 5 · 2022: 5 · 2021: 2 · 2020: 6 · 2019: 12 · 2018: 9 · 2017: 14 · 2016: 18 · 2015: 19 · 2014: 29 · 2013: 37 · 2012: 32
- FWCI
- 30.05
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) (publishedVersion)
- Funders
- Heinz Endowments
- Intel Corporation
- Microsoft Research
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Mobile Crowdsensing and Crowdsourcing
- Social Sciences › Psychology › Developmental and Educational Psychology › Educational Games and Gamification
- Social Sciences › Decision Sciences › Management Science and Operations Research › Game Theory and Applications
- Keywords
- games with a purpose, crowdsourcing, music tagging, object identification, collaborative games
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review