Cited by 169Music and Audio ProcessingRecommender Systems and TechniquesMusic Technology and Sound Studies

Evaluation Of Algorithms Using Games: The Case Of Music Tagging.

Open access
Law, Edith, West, Kris, Mandel, Michael I., Bay, Mert, J. Stephen Downie·2009·Zenodo

Abstract

Search by keyword is an extremely popular method for retrieving music. To support this, novel algorithms that automatically tag music are being developed. The conventional way to evaluate audio tagging algorithms is to compute measures of agreement between the output and the ground truth set. In this work, we introduce a new method for evaluating audio tagging algorithms on a large scale by collecting set-level judgments from players of a human computation game called TagATune. We present the design and preliminary results of an experiment comparing five algorithms using this new evaluation metric, and contrast the results with those obtained by applying several conventional agreement-based evaluation metrics. 1.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.5281/zenodo.1417646

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The models are trained with the MagnaTagATune dataset (19k training songs) and the Million Song Dataset (200k training songs).

    “The MTT models are trained with the MagnaTagATune dataset, and the MSD models are trained with the Million Song Dataset.”

    From musicnn: Pre-trained convolutional neural networks for music audio tagging (2019) · cites this paper as [3]

    Section 2.1.1, “The Dataset” (p. 388), says the participating algorithms were trained on a subset of the TagATune dataset. It reports 16,289 training music clips, not 19,000 songs, and does not address models trained on the Million Song Dataset or a 200,000-song training set. (LLM, conf=0.87)

    Experimental — accuracy not yet measured
  • MTT consists of 26k music clips from 5,223 unique songs including genre, instrument, vocal, mood, perceptual tempo, origin, and sonority features.

    “MTT [31] consists of 26k music clips from 5,223 unique songs including genre, instrument, vocal, mood, perceptual tempo, origin, and sonority features.”

    From LP-MusicCaps: LLM-Based Pseudo Music Captioning (2023) · cites this paper as [31]

    Section 2.1.1, “The Dataset” (p. 388), describes a subset of the TagATune dataset, not MTT. Searches for “MTT,” “5,223,” and the listed feature categories found no passage in our copy establishing MTT’s size or contents. (LLM, conf=0.76)

    Experimental — accuracy not yet measured
  • Prevailing methodologies for contemporary musical descriptions typically lean on discriminative learning.

    “In the realm of Music Information Retrieval (MIR), prevailing methodologies for contemporary musical descriptions typically lean on discriminative learning.”

    From MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response (2024) · cites this paper as Law et al., 2009

    The Introduction (p. 387) describes music-tagging algorithms and conventional agreement-based evaluation, but does not say that prevailing methods rely on discriminative learning. Searches for “discriminative learning” and related descriptions of classification methods found no passage establishing that claim. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Law, E., West, K., Mandel, M. I., Bay, M., & Downie, J. S.
Affiliations
  • Carnegie Mellon University (US)
  • University of Illinois Urbana-Champaign (US)
  • Columbia University (US)
Venue
Zenodo
Publisher
Zenodo
Published
2009-10-26
Pages
387-392
OpenAlex
W2127870748
Cited by
168
References
3
Citations by year
2026: 1 · 2025: 10 · 2024: 20 · 2023: 15 · 2022: 17 · 2021: 15 · 2020: 22 · 2019: 13 · 2018: 11 · 2017: 9 · 2016: 8 · 2015: 1 · 2014: 4 · 2013: 6 · 2012: 3
FWCI
8.40
Type
journal article
Language
en
Open access
Licence
other-oa
Locations
  • Zenodo (CERN European Organization for Nuclear Research) (submittedVersion, other-oa)
  • Zenodo (CERN European Organization for Nuclear Research) (other-oa)
  • International Symposium/Conference on Music Information Retrieval
Topics
  • Physical Sciences › Computer Science › Signal Processing › Music and Audio Processing
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Music Technology and Sound Studies
Keywords
music tagging, algorithm evaluation, crowdsourcing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review