Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 313

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation

Nowak, Stefanie, Rüger, Stefan·2010·Proceedings of the international conference on Multimedia information retrieval
DOI 10.1145/1743384.1743478
Open this paper in CoChat — free account

Abstract

The creation of golden standard datasets is a costly business. Optimally more than one judgment per document is obtained to ensure a high quality on annotations. In this context, we explore how much annotations from experts differ from each other, how different sets of annotations influence the ranking of systems and if these annotations can be obtained with a crowdsourcing approach. This study is applied to annotations of images with multiple concepts. A subset of the images employed in the latest ImageCLEF Photo Annotation competition was manually annotated by expert annotators and non-experts with Mechanical Turk. The inter-annotator agreement is computed at an image-based and concept-based level using majority vote, accuracy and kappa statistics. Further, the Kendall τ and Kolmogorov-Smirnov correlation test is used to compare the ranking of systems regarding different ground-truths and different evaluation measures in a benchmark scenario. Results show that while the agreement between experts and non-experts varies depending on the measure used, its influence on the ranked lists of the systems is rather small. To sum up, the majority vote applied to generate one annotation set out of several opinions, is able to filter noisy judgments of non-experts to some extent. The resulting annotation set is of comparable quality to the annotations of experts.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/1743384.1743478

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 1 references

Research works printed but not linked: 26

  1. O. Alonso and S. Mizzaro . Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment . In SIGIR 2009 Workshop on the Future of IR Evaluation. O. Alonso and S. Mizzaro. Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment. In SIGIR 2009 Workshop on the Future of IR Evaluation.
  2. T. Brants . Inter-annotator agreement for a German newspaper corpus . In Proc. of the 2nd Intern. Conf. on Language Resources and Evaluation , 2000 . T. Brants. Inter-annotator agreement for a German newspaper corpus. In Proc. of the 2nd Intern. Conf. on Language Resources and Evaluation, 2000.
  3. 10.1177/001316448104100307 · DOI · Paper page
  4. 10.1145/1631272.1631339 · DOI
  5. T. Chklovski and R. Mihalcea . Exploiting agreement and disagreement of human annotators for word sense disambiguation . In Proceedings of RANLP 2003 , 2003 . T. Chklovski and R. Mihalcea. Exploiting agreement and disagreement of human annotators for word sense disambiguation. In Proceedings of RANLP 2003, 2003.
  6. J. Cohen A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37--46 , 1960 . J. Cohen et al. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37--46, 1960.
  7. 10.1145/1557019.1557053 · DOI
  8. D. Feng and S. Zajac . Acquiring High Quality Non-Expert Knowledge from On-demand Workforce. ACL-IJCNLP 2009 . D. Feng and S. Zajac. Acquiring High Quality Non-Expert Knowledge from On-demand Workforce. ACL-IJCNLP 2009.
  9. J. Howe . Crowdsourcing - A Definition . http://crowdsourcing.typepad.com/cs/2006/06/, last accessed 24.11.2009 , 2006 . J. Howe. Crowdsourcing - A Definition. http://crowdsourcing.typepad.com/cs/2006/06/, last accessed 24.11.2009, 2006.
  10. 10.5555/1564131.1564137 · doi:10.5555/1564131.1564137
  11. 10.1145/1460096.1460104 · DOI · Paper page
  12. G. Kazai and N. Milic-Frayling . On the Evaluation of the Quality of Relevance Assessments Collected through Crowdsourcing . In SIGIR 2009 Workshop on the Future of IR Evaluation. G. Kazai and N. Milic-Frayling. On the Evaluation of the Quality of Relevance Assessments Collected through Crowdsourcing. In SIGIR 2009 Workshop on the Future of IR Evaluation.
  13. 10.1093/biomet/30.1-2.81 · DOI · Paper page
  14. 10.1006/csla.1998.0108 · doi:10.1006/csla.1998.0108
  15. A. Kolmogoroff . Sulla determinazione empirica di una legge di distribuzione . Giorn. Ist. Ital. Attuari , 4 ( 1 ): 83 -- 91 , 1933 . A. Kolmogoroff. Sulla determinazione empirica di una legge di distribuzione. Giorn. Ist. Ital. Attuari, 4(1):83--91, 1933.
  16. 10.2307/2529310 · DOI · Paper page
  17. S. Nowak and P. Dunker . A Consumer Photo Tagging Ontology: Concepts and Annotations . In THESEUS-ImageCLEF Pre-Workshop 2009 , Corfu, Greece , 2009 . S. Nowak and P. Dunker. A Consumer Photo Tagging Ontology: Concepts and Annotations. In THESEUS-ImageCLEF Pre-Workshop 2009, Corfu, Greece, 2009.
  18. S. Nowak and P. Dunker . Overview of the CLEF 2009 Large-Scale Visual Concept Detection and Annotation Task. CLEF working notes 2009 , Corfu, Greece , 2009 . S. Nowak and P. Dunker. Overview of the CLEF 2009 Large-Scale Visual Concept Detection and Annotation Task. CLEF working notes 2009, Corfu, Greece, 2009.
  19. S. Nowak and H. Lukashevich . Multilabel Classification Evaluation using Ontology Information . In Proc. of IRMLeS Workshop, ESWC , Greece , 2009 . S. Nowak and H. Lukashevich. Multilabel Classification Evaluation using Ontology Information. In Proc. of IRMLeS Workshop, ESWC, Greece, 2009.
  20. J. Randolph . Free-Marginal Multirater Kappa (multirater κ free): An Alternative to Fleiss' Fixed-Marginal Multirater Kappa . In Joensuu Learning and Instruction Symposium , 2005 . J. Randolph. Free-Marginal Multirater Kappa (multirater κ free): An Alternative to Fleiss' Fixed-Marginal Multirater Kappa. In Joensuu Learning and Instruction Symposium, 2005.
  21. 10.1145/1401890.1401965 · DOI · Paper page
  22. 10.5555/1613715.1613751 · doi:10.5555/1613715.1613751
  23. 10.1109/CVPRW.2008.4562953 · DOI · Paper page
  24. J. Véronis . A study of polysemy judgements and inter-annotator agreement . In Programme and advanced papers of the Senseval workshop , 1998 . J. Véronis. A study of polysemy judgements and inter-annotator agreement. In Programme and advanced papers of the Senseval workshop, 1998.
  25. 10.1109/TMM.2007.900153 · doi:10.1109/tmm.2007.900153
  26. 10.1016/S0306-4573(00)00010-8 · DOI · Paper page

Cited by

Metadata

Authors
Nowak, S., & Rüger, S.
Venue
Proceedings of the international conference on Multimedia information retrieval
Publisher
ACM
Published
2010-03-29
Pages
557-566
DOI
10.1145/1743384.1743478
OpenAlex
W2003497265
Cited by
313
References
1
FWCI
18.42
Type
conference paper
Language
en
Open access
closed
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.