The DOI matches its DataCite record; 3 of the 3 references it cites were found in a registry or an index.
Evaluation Of Algorithms Using Games: The Case Of Music Tagging.
Open accessAbstract
Search by keyword is an extremely popular method for retrieving music. To support this, novel algorithms that automatically tag music are being developed. The conventional way to evaluate audio tagging algorithms is to compute measures of agreement between the output and the ground truth set. In this work, we introduce a new method for evaluating audio tagging algorithms on a large scale by collecting set-level judgments from players of a human computation game called TagATune. We present the design and preliminary results of an experiment comparing five algorithms using this new evaluation metric, and contrast the results with those obtained by applying several conventional agreement-based evaluation metrics. 1.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The MTT models are trained with the MagnaTagATune dataset, and the MSD models are trained with the Million Song Dataset.”
From musicnn: Pre-trained convolutional neural networks for music audio tagging (2019) · cites this paper as [3]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“MTT [31] consists of 26k music clips from 5,223 unique songs including genre, instrument, vocal, mood, perceptual tempo, origin, and sonority features.”
From LP-MusicCaps: LLM-Based Pseudo Music Captioning (2023) · cites this paper as [31]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“In the realm of Music Information Retrieval (MIR), prevailing methodologies for contemporary musical descriptions typically lean on discriminative learning.”
From MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response (2024) · cites this paper as Law et al., 2009
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Downie, J. S. (2008) · Nippon Onkyo Gakkaishi/Acoustical science and technology/Nihon Onkyo Gakkaishi · doi:10.1250/ast.29.247 · Cited by 297
Self-citationA Web-Based Game for Collecting Music Metadata
Mandel, M., & Ellis, D. P. W. (2008) · Journal of New Music Research · doi:10.1080/09298210802479300 · Cited by 129
Self-citationA Game-Based Approach For Collecting Semantic Annotations Of Music.
Turnbull, D., Liu, R., Barrington, L., & Lanckriet, G. R. G. (2007) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1416463 · Cited by 98
Flagged first, then by citations
Cited by
Sorted by citations
Items 101–169 of 169
Cho, S.-H., Park, Y., & Lee, J. (2022) · IEEE International Conference on Consumer Electronics-Asia (ICCE-Asia) · doi:10.1109/icce-asia57006.2022.9954732 · Cited by 5
Feature-informed Embedding Space Regularization For Audio Classification
Hung, Y.-N. A., & Lerch, A. (2022) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco55093.2022.9909904 · Cited by 6
McCallum, M. C., Davies, M. E. P., Henkel, F., Kim, J., & Sandberg, S. E. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446274 · Cited by 9
MediaEval 2019 Emotion and Theme Recognition task: A VQ-VAE Based Approach.
Hung, H.-T., Chen, Y., Mayerl, M., Vötter, M., Zangerle, E., & Yang, Y. (2019) · MediaEval · Cited by 4
S3T: Self-Supervised Pre-training with Swin Transformer for Music ClassificationPreprint
Zhao, H., Zhang, C., Zhu, B., Ma, Z., & Zhang, K. (2022) · doi:10.31219/osf.io/buwcv · Cited by 5
LP-MusicCaps: LLM-Based Pseudo Music Captioning
Doh, S., Choi, K., Lee, J., & Nam, J. (2023) · arXiv · doi:10.48550/arxiv.2307.16372 · Cited by 16
Source Separation By Steering Pretrained Music Models
Manilow, E., O’Reilly, P., Seetharaman, P., & Pardo, B. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747909 · Cited by 6
On the Role of Visual Context in Enriching Music Representations
Avramidis, K., Stewart, S., & Narayanan, S. S. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094915 · Cited by 5
Seethevoice: Learning from Music to Visual Storytelling of Shots
Wei, W.-L., Lin, J., Liu, T.-L., Yang, Y., Wang, H., Tyan, H.-R., & Liao, H.-Y. M. (2018) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2018.8486496 · Cited by 5
Fonts That Fit the Music: A Multimodal Design Trend Analysis of Lyric Videos
Haraguchi, D., Sakaguchi, S., Kato, J., Goto, M., & Uchida, S. (2022) · IEEE Access · doi:10.1109/access.2022.3184028 · Cited by 4
Music Question Answering:Cognize and Perceive Music
Gao, W., Li, X., Jin, C., & Tie, Y. (2022) · IEEE International Conference on Multimedia and Expo workshops · doi:10.1109/icmew56448.2022.9859499 · Cited by 4
Generating Multimodal Augmentations with LLMs from Song Metadata for Music Information Retrieval
Rossetto, F., Dalton, J., & Murray‐Smith, R. (2023) · doi:10.1145/3607827.3616842 · Cited by 4
Similar but Faster: Manipulation of Tempo in Music Audio Embeddings for Tempo Prediction and Search
McCallum, M. C., Henkel, F., Kim, J., Sandberg, S. E., & Davies, M. E. P. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447877 · Cited by 5
Benchmarking Music Generation Models and Metrics via Human Preference Studies
Grötschla, F., Solak, A., Lanzendörfer, L. A., & Wattenhofer, R. P. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887745 · Cited by 6
Guinot, J., Quinton, E., & Fazekas, G. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10890270 · Cited by 3
Hacker's Paradise: Analysing Music in a Cybercrime Forum
Talas, A., & Hutchings, A. (2023) · APWG Symposium on Electronic Crime Research (eCrime) · doi:10.1109/ecrime61234.2023.10485503 · Cited by 3
Transfer Learning for Violinist Identification
Zhao, Y., Fazekas, G., & Sandler, M. B. (2022) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco55093.2022.9909590 · Cited by 3
Music Auto-Tagging with Capsule Network
Yu, Y., Tang, Y., Qi, M., Mai, F., Deng, Q., & Zhaxi, N. (2020) · Communications in computer and information science · doi:10.1007/978-981-15-7981-3_20 · Cited by 3
Deep Learning Based EDM Subgenre Classification using Mel-Spectrogram and Tempogram FeaturesPreprint
Hsu, W.-H., Chen, B., & Yang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08862 · Cited by 8
A Method for Assessing User-generated Tests for Online Courses Exploiting Crowdsourcing Concept
Elsayed, G., & Inoue, S. (2014) · International Workshop on Web Intelligence and Smart Sensing (IWWISS) · doi:10.1145/2637064.2637102 · Cited by 2
Unsupervised Selection of Robust Audio Feature Subsets
Sageder, G., Zaharieva, M., & Zeppelzauer, M. (2014) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611973440.79 · Cited by 2
Bayle, Y., Hanna, P., & Robine, M. (2017) · International Workshop on Content-Based Multimedia Indexing (CBMI) · doi:10.1145/3095713.3095716 · Cited by 2
Pilot Case Study in Games as Polling Systems, Generating Knowledge about Fake News
Grace, L. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3375230 · Cited by 2
A Modulation Front-End for Music Audio Tagging
Vahidi, C., Saitis, C., & Fazekas, G. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533547 · Cited by 2
A Content Analysis of the Research Approaches in Music Genre Recognition
Özseven, T., & Ozseven, B. E. (2022) · International Congress on Human-Computer Interaction, Optimization and Robotic Applications (ICHORA) · doi:10.1109/hora55278.2022.9799935 · Cited by 2
Affinity Weighted EmbeddingPreprint
Weston, J., Weiss, R. J., & Yee, H. Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.4171 · Cited by 5
Towards Learning Universal Audio RepresentationsPreprint
Wang, L., Luc, P., Wu, Y. X., Recasens, A., Smaira, L., Brock, A., Jaegle, A., Alayrac, J.-B., Dieleman, S., Carreira, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12124 · Cited by 2
METER2800: A novel dataset for music time signature detection
Abimbola, J. O., Kostrzewa, D., & Kasprowski, P. (2023) · Data in Brief · doi:10.1016/j.dib.2023.109736 · Cited by 2
ATOSE: Audio Tagging with One-Sided Joint Embedding
Lee, J., Moon, D., Kim, J., & Cho, M. (2023) · Applied Sciences · doi:10.3390/app13159002 · Cited by 1
VICMus: Variance-Invariance-Covariance Regularization for Music Representation Learning
Löf, S., Hesse, C., Thomé, C., Lordelo, C., & Ahrens, J. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10627508 · Cited by 1
Learning Music Audio Representations With Limited Data
Plachouras, C., Benetos, E., & Pauwels, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887766 · Cited by 1
Evaluating Contrastive Methodologies for Music Representation Learning Using Playlist Data
Meehan, G., & Pauwels, J. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888157 · Cited by 2
Keep what you need : extracting efficient subnetworks from large audio representation models
Genova, D., Esling, P., & Hurlin, T. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887836 · Cited by 1
MGPHot: A Dataset of Musicological Annotations for Popular Music (1958–2022)
Oramas, S., Gouyon, F., Hogan, S., Landau, C., & Ehmann, A. F. (2025) · Transactions of the International Society for Music Information Retrieval · doi:10.5334/tismir.236 · Cited by 1
MusiQAl: A Dataset for Music Question–Answering through Audio–Video Fusion
Christodoulou, A.-M., Glette, K., Lartillot, O., & Jensenius, A. R. (2025) · Transactions of the International Society for Music Information Retrieval · doi:10.5334/tismir.222 · Cited by 5
Music Auto-Tagging with Robust Music Representation Learned via Domain Adversarial Training
Joung, H., & Lee, K. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447318 · Cited by 1
Embedding Compression for Teacher-to-Student Knowledge Transfer
Ding, Y., & Lerch, A. (2024) · IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) · doi:10.1109/icasspw62465.2024.10626662 · Cited by 1
LC-Protonets: Multi-Label Few-Shot Learning for World Music Audio Tagging
Papaioannou, C., Benetos, E., & Potamianos, A. (2025) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2025.3529315 · Cited by 1
Towards Training Music Taggers on Synthetic Data
Kroher, N., Manangu, S., & Pikrakis, A. (2024) · International Workshop on Content-Based Multimedia Indexing (CBMI) · doi:10.1109/cbmi62980.2024.10859229 · Cited by 2
Subtractive Training for Music Stem Insertion Using Latent Diffusion Models
Villa-Renteria, I., Wang, M. L., Shah, Z., Li, Z., Kim, S.-H., Ramachandran, N., & Pilancı, M. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10887744 · Cited by 1
The Influence of Crowdsourcing Business Model into Artificial Intelligence
Szopa, A. (2019) · IGI Global eBooks · doi:10.4018/978-1-5225-8362-2.ch013 · Cited by 1
Multi-scale Embedded CNN for Music Tagging (MsE-CNN)Preprint
Hamidi, N., Vahidzadeh, M., & Baek, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.06746 · Cited by 1
Learning Contextualized Music Semantics from Tags via a Siamese NetworkPreprint
Sandouk, U., & Chen, K. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.07968 · Cited by 1
Retrieval and Annotation of Music Using Latent Semantic Models
Lévy, M. (2012) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 1
The Influence of Crowdsourcing Business Model into Artificial Intelligence
Szopa, A. (2016) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-5225-1656-9.ch002
Audio-based Distributional Semantic Models for Music Auto-tagging and Similarity MeasurementPreprint
Karamanolakis, G., Iosif, E., Zlatintsi, A., Pikrakis, A., & Potamianos, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08391
Fusion multi-niveaux par boosting pour le tagging automatique
Gaël Richard, A., Essid, S., & Lagrange, M. (2013)
Source-separated audio input for accelerating convolutional neural networks
Domínguez, M., Daigneau, M., & Ptucha, R. (2017) · IEEE Western New York Image and Signal Processing Workshop (WNYISPW) · doi:10.1109/wnyipw.2017.8356323
Comparative Study on Different Approaches in Optimizing Threshold for Music Auto-Tagging
Minh, K. N. C., An, T. D., Quang, V. T., & Tran, V. H. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-03192-3_18
A Multi-scale Convolutional Neural Network Architecture for Music Auto-Tagging
Dabral, T. S., Deshmukh, A. S., & Malapati, A. (2018) · Advances in intelligent systems and computing · doi:10.1007/978-981-13-1592-3_60
Middlebrook, K., Sudhakaran, S., & Brizan, D. G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.02309
Unsupervised Source Separation By Steering Pretrained Music ModelsPreprint
Manilow, E., O’Reilly, P., Seetharaman, P., & Pardo, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.13071 · Cited by 3
Progressive Teacher-Student Training Framework for Music Tagging
Lü, R., Zheng, B., Hai, J., Tao, F., Duan, Z., & Liu, J. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747342
The Influence of Crowdsourcing Business Model Into Artificial Intelligence
Szopa, A. (2019) · IGI Global eBooks · doi:10.4018/978-1-5225-8182-6.ch027
Implementation of Computationally Efficient And Accurate Music Auto Tagging
Rajendran, S., & Anandaraj, S. P. (2022) · International Conference on Advanced Computing and Communication Systems (ICACCS) · doi:10.1109/icaccs54159.2022.9785061
Deep Chnet — 1-D CNN using raw audio waveform for recognizing traditional Chinese musical instrument
Wang, B. (2023) · Applied and Computational Engineering · doi:10.54254/2755-2721/6/20220802
Music Question Answering Based on Aesthetic Experience
Gao, W., Li, X., Tie, Y., & Qi, L. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191775
A Novel Method to Detect Instrumental Music in a Large Scale Music Catalog
Lee, W. J., & Coviello, E. (2023) · IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) · doi:10.1109/waspaa58266.2023.10248164
Auxiliary self-supervision to metric learning for music similarity-based retrieval and auto-tagging
Akama, T., Kitano, H., Takematsu, K., Miyajima, Y., & Polouliakh, N. (2023) · PLoS ONE · doi:10.1371/journal.pone.0294643
WikiMuTe: A web-sourced dataset of semantic descriptions for music audioPreprint
Weck, B., Kirchhoff, H., Grosche, P., & Serra, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.09207
MusiQAl: Music Question-Answering through Audio-Video FusionPreprint
Christodoulou, A.-M., Glette, K., Lartillot, O., & Jensenius, A. R. (2024) · doi:10.31219/osf.io/2vy3p
Playlist Continuation of Cold-Start Songs
Cheng, C.-H., Wei, T.-Y., & Chen, H. H. (2024) · IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR) · doi:10.1109/mipr62202.2024.00029
MuIm: Analyzing Music–Image Correlations from an Artistic Perspective
Ullah, U., & Choi, H. (2024) · Applied Sciences · doi:10.3390/app142311470
Algorithmes de recommandation musicale
Maillet, F. (2009) · doi:10.71781/10290
Probing for Advanced Music Theory Concepts in Generative Music Models
Kwan, D., & Donnelly, P. J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-24350-8_17
SonicVerse: Multi-Task Learning for Music Feature-Informed Captioning
Chopra, A., Roy, A., & Herremans, D. (2025) · arXiv · doi:10.48550/arxiv.2506.15154 · Cited by 3
Metadata
- Authors
- Law, E., West, K., Mandel, M. I., Bay, M., & Downie, J. S.
- Affiliations
- Carnegie Mellon University (US)
- University of Illinois Urbana-Champaign (US)
- Columbia University (US)
- Venue
- Zenodo
- Publisher
- Zenodo
- Published
- 2009-10-26
- Pages
- 387-392
- OpenAlex
- W2127870748
- Cited by
- 168
- References
- 3
- Citations by year
- 2026: 1 · 2025: 10 · 2024: 20 · 2023: 15 · 2022: 17 · 2021: 15 · 2020: 22 · 2019: 13 · 2018: 11 · 2017: 9 · 2016: 8 · 2015: 1 · 2014: 4 · 2013: 6 · 2012: 3
- FWCI
- 8.40
- Type
- journal article
- Language
- en
- Open access
- green
- zenodo.org
- Licence
- other-oa
- Locations
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, other-oa)
- Zenodo (CERN European Organization for Nuclear Research) (other-oa)
- International Symposium/Conference on Music Information Retrieval
- Topics
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Social Sciences › Arts and Humanities › Music › Music Technology and Sound Studies
- Keywords
- music tagging, algorithm evaluation, crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
