The DOI matches its DataCite record; 3 of the 3 references it cites were found in a registry or an index.
Evaluation Of Algorithms Using Games: The Case Of Music Tagging.
Open accessAbstract
Search by keyword is an extremely popular method for retrieving music. To support this, novel algorithms that automatically tag music are being developed. The conventional way to evaluate audio tagging algorithms is to compute measures of agreement between the output and the ground truth set. In this work, we introduce a new method for evaluating audio tagging algorithms on a large scale by collecting set-level judgments from players of a human computation game called TagATune. We present the design and preliminary results of an experiment comparing five algorithms using this new evaluation metric, and contrast the results with those obtained by applying several conventional agreement-based evaluation metrics. 1.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The MTT models are trained with the MagnaTagATune dataset, and the MSD models are trained with the Million Song Dataset.”
From musicnn: Pre-trained convolutional neural networks for music audio tagging (2019) · cites this paper as [3]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“MTT [31] consists of 26k music clips from 5,223 unique songs including genre, instrument, vocal, mood, perceptual tempo, origin, and sonority features.”
From LP-MusicCaps: LLM-Based Pseudo Music Captioning (2023) · cites this paper as [31]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“In the realm of Music Information Retrieval (MIR), prevailing methodologies for contemporary musical descriptions typically lean on discriminative learning.”
From MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response (2024) · cites this paper as Law et al., 2009
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Downie, J. S. (2008) · Nippon Onkyo Gakkaishi/Acoustical science and technology/Nihon Onkyo Gakkaishi · doi:10.1250/ast.29.247 · Cited by 297
Self-citationA Web-Based Game for Collecting Music Metadata
Mandel, M., & Ellis, D. P. W. (2008) · Journal of New Music Research · doi:10.1080/09298210802479300 · Cited by 129
Self-citationA Game-Based Approach For Collecting Semantic Annotations Of Music.
Turnbull, D., Liu, R., Barrington, L., & Lanckriet, G. R. G. (2007) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1416463 · Cited by 98
Cited by
Sorted by citations
Quinn, A. J., & Bederson, B. B. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979148 · Cited by 837
SoundStream: An End-to-End Neural Audio Codec
Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., & Tagliasacchi, M. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3129994 · Cited by 637
FMA: A Dataset for Music Analysis
Defferrard, M., Benzi, K., Vandergheynst, P., & Bresson, X. (2016) · arXiv · doi:10.48550/arxiv.1612.01840 · Cited by 237
Lee, J., & Nam, J. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2017.2713830 · Cited by 155
Lee, J., Park, J., Kim, K., & Nam, J. (2018) · Applied Sciences · doi:10.3390/app8010150 · Cited by 136
Metadata
- Authors
- Law, E., West, K., Mandel, M. I., Bay, M., & Downie, J. S.
- Affiliations
- Carnegie Mellon University (US)
- University of Illinois Urbana-Champaign (US)
- Columbia University (US)
- Venue
- Zenodo
- Publisher
- Zenodo
- Published
- 2009-10-26
- Pages
- 387-392
- OpenAlex
- W2127870748
- Cited by
- 168
- References
- 3
- Citations by year
- 2026: 1 · 2025: 10 · 2024: 20 · 2023: 15 · 2022: 17 · 2021: 15 · 2020: 22 · 2019: 13 · 2018: 11 · 2017: 9 · 2016: 8 · 2015: 1 · 2014: 4 · 2013: 6 · 2012: 3
- FWCI
- 8.40
- Type
- journal article
- Language
- en
- Open access
- green
- zenodo.org
- Licence
- other-oa
- Locations
- Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, other-oa)
- Zenodo (CERN European Organization for Nuclear Research) (other-oa)
- International Symposium/Conference on Music Information Retrieval
- Topics
- Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Social Sciences › Arts and Humanities › Music › Music Technology and Sound Studies
- Keywords
- music tagging, algorithm evaluation, crowdsourcing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
