CoChat
Sign Up
Cited by 34Natural Language Processing TechniquesTopic ModelingSentiment Analysis and Opinion Mining

The DOI matches its Crossref record.

Multiplicity and word sense: evaluating and learning from multiply labeled word sense annotations

Passonneau, Rebecca J., Bhardwaj, Vikas, Salleb-Aouissi, Ansaf, Ide, Nancy·2012·Language Resources and Evaluation
DOI 10.1007/s10579-012-9188-x
Open this paper in CoChat — free account

Overview

Integrity signals

Registry

Crossref · 10.1007/s10579-012-9188-x

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There is no correlation between agreement and number of available senses, or senses used by annotators.

    “There is no correlation between the agreement and number of available senses, or senses used by annotators (Passonneau et al., 2012a).”

    From The Benefits of a Model of Annotation (2014) · cites this paper as Passonneau et al., 2012a

  • The MASC word sense data demonstrates that annotators can agree on words with many senses, but there are many words with low agreement and correspondingly questionable ground truth labels.

    “While the MASC word sense data demonstrates that annotators can agree on words with many senses, there are many words with low agreement, and correspondingly questionable ground truth labels.”

    From The Benefits of a Model of Annotation (2014) · cites this paper as Passonneau et al., 2012a

  • Chance-adjusted agreement levels for MASC word sense annotations ranged from very high to chance levels, with similar variation for pairwise agreement.

    “Chanceadjusted agreement levels ranged from very high to chance levels, with similar variation for pairwise agreement (Passonneau et al., 2012a).”

    From The Benefits of a Model of Annotation (2014) · cites this paper as Passonneau et al., 2012a

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 52

  1. Agirre, E., de Lacalle, O. L., Fellbaum, C., Hsieh, S. K., Tesconi, M., Monachini, M., Vossen, P., & Segers, R. (2010). SemEval-2010 Task 17: All-words word sense disambiguation on a specific domain. In Proceedings of the 5th international workshop on semantic evaluation (pp. 75–80).
  2. Akkaya, C., Conrad, A., Wiebe, J., & Mihalcea, R. (2010). Amazon Mechanical Turk for subjectivity word sense disambiguation. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s Mechanical Turk, Association for Computational Linguistics, Los Angeles (pp. 195–203).
  3. Bhardwaj, V., Passonneau, R. J., Salleb-Aouissi, A., & Ide, N. (2010). Anveshan: A framework for analysis of multiple annotators’ labeling behavior. In Proceedings of the fourth linguistic annotation workshop (LAW IV).
  4. Bruce, R. F., & Wiebe, J. M. (1999). Decomposable modeling in natural language processing. Computational Linguistics, 25(2), 195-208.
  5. Callison-Burch, C. (2009). Fast, cheap, and creative: evaluating translation quality using Amazon’s Mechanical Turk. In Proceedings of the 2009 conference on empirical methods in natural language processing, Association for Computational Linguistics, Morristown, NJ (pp. 286–295). · DOI · Paper page
  6. Callison-Burch, C., & Dredze, M. (2010). Creating speech and language data with Amazon’s Mechanical Turk. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s Mechanical Turk (pp. 1–12).
  7. Chugur, I., Gonzalo, J., & Verdejo, F. (2002). Polysemy and sense proximity in the SENSEVAL-2 test suite. In Proceedings of the SIGLEX/SENSEVAL workshop on word sense disambiguation: Recent successes and future directions, Philadelphia (pp. 32–39). · DOI
  8. Cohen, J. (1960). A coeffiecient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46. · DOI · Paper page
  9. Diab, M. (2004). Relieving the data acquisition bottleneck in word sense disambiguation. In Proceedings of the 42nd annual meeting on association for computational linguistics (pp. 303–311). · DOI
  10. Dowty, D. (1979). Word meaning and montague grammar. Dordrecht: D. Reidel. · DOI
  11. Erk, K. (2009). Representing words as regions in vector space. In CoNLL ’09: Proceedings of the 13th conference on computational natural language learning (pp. 57–65). · DOI
  12. Erk, K., & Mccarthy, D. (2009). Graded word sense assignment. In Proceedings of empirical methods in natural language processing (EMNLP 09) (pp. 440–449). · DOI
  13. Erk, K., McCarthy, D., & Gaylord, N. (2009). Investigations on word senses and word usages. In Proceedings of the 47th annual meeting of the association for computational linguistics and the 4th international joint conference on natural language processing (pp. 10–18). · DOI
  14. Fillmore, C. J., Johnson, C. R., & Petruck, M. R. L. (2003). Background to framenet. International Journal of Lexicography, 16(3), 235–250. · DOI
  15. Hovy, E., Marcus, M., Palmer, M., Ramsha, L., & Weischedel, R. (2006). Ontonotes: The 90% solution. In Proceedings of HLT-NAACL 2006 (pp. 57–60). · DOI
  16. Ide, N. (2000). Cross-lingual sense determination: Can it work? Computers and the Humanities. Special Issue on the proceedings of the SIGLEX/SENSEVAL Workshop, 34(1–2), 223–234.
  17. Ide, N., & Wilks, Y. (2006). Making sense about sense. In E. Agirre & P. Edmonds (Eds.), Word sense disambiguation: Algorithms and applications (pp. 47–74). Dordrecht: Springer. · DOI
  18. Ide, N., Erjavec, T., & Tufis, D. (2002). Sense discrimination with parallel corpora. In Proceedings of ACL’02 workshop on word sense disambiguation: Recent successes and future directions (pp. 54–60). · DOI
  19. Ide, N., Baker, C., Fellbaum, C., & Passonneau, R. J. (2010). The manually annotated sub-corpus: A community resource for and by the people. In Proceedings of the association for computational linguistics (pp. 68–73).
  20. Kilgarriff, A. (1997). I don’t believe in word senses. Computers and the Humanities, 31, 91–113. · DOI
  21. Kilgarriff, A. (1998). SENSEVAL: An exercise in evaluating word sense disambiguation programs. In Proceedings of the 1st international conference on language resources and evaluation (LREC), Granada (pp. 581–588).
  22. Klein, D., & Murphy, G. (2002). Paper has been my ruin: Conceptual relations of polysemous words. Journal of Memory and Language, 47, 548. · DOI
  23. Krippendorff, K. (1980). Content analysis: An introduction to its methodology. Beverly Hills, CA: Sage Publications.
  24. Kullback, S., & Leibler, R. A. (1951). On information and sufficiency. Annals of Mathematical Statistics, 22(1), 79–86. · DOI · Paper page
  25. Landauer, T., & Dumais, S. (1977). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211–240. · DOI · Paper page
  26. Lavrac, N., Flach, P. A., & Zupan, B. (1999). Rule evaluation measures: a unifying view. In Proceedings of the 9th international workshop on inductive logic programming (ILP-99) (pp. 174–185). · DOI
  27. Lin, J. (1991). Divergence measures based on the Shannon entropy. IEEE Transactions on Information Theory, 37(1), 145–151. · DOI
  28. Manandhar, S., Klapaftis, I., Dligach, D., & Pradhan, S. (2010). SemEval-2010 task 14: Word sense induction & disambiguation. In Proceedings of the 5th international workshop on semantic evaluation (SemEval), Association for Computational Linguistics, Uppsala, Sweden (pp. 63–68).
  29. Navigli, R. (2009). Word sense disambiguation: A survey. ACM Computing Surveys, 41(2), 10:1–10:69. · DOI
  30. Ng, H. T., Lim, C. Y., & Foo, S. K. (1999). A case study on inter-annotator agreement for word sense disambiguation. In SIGLEX Workshop On Standardizing Lexical Resources.
  31. Palmer, M., Dang, H. T., & Fellbaum, C. (2007). Making fine-grained and coarse-grained sense distinctions, both manually and automatically. Natural Language Engineering, 13(2), 137–163. · DOI
  32. Passonneau, R. J. (2006). Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. In Fifth international conference on language resources and evaluation (LREC).
  33. Passonneau, R. J., Habash, N., & Rambow, O. (2006). Inter-annotator agreement on a multilingual semantic annotation task. In Proceedings of the international conference on language resources and evaluation (LREC), Genoa, Italy (pp. 1951–1956).
  34. Passonneau, R. J., Salleb-Aouissi, A., & Ide, N. (2009). Making sense of word sense variation. In Proceedings of the NAACL-HLT 2009 workshop on semantic evaluations. · DOI
  35. Passonneau, R. J., Salleb-Aouissi, A., Bhardwaj, V., & Ide, N. (2010). Word sense annotation of polysemous words by multiple annotators. In Seventh international conference on language resources and evaluation (LREC).
  36. Passonneau, R. J., Baker, C., Fellbaum, C., & Ide, N. (2012). The MASC word sense sentence corpus. In Proceedings of the 8th international conference on language resources and evaluation (LREC), Istanbul, Turkey, May 23–25.
  37. Pedersen, T. (2002a). Assessing system agreement and instance difficulty in the lexical sample tasks of SENSEVAL-2. In Proceedings of the ACL-02 workshop on word sense disambiguation: Recent successes and future directions (pp. 40–46). · DOI
  38. Pedersen, T. (2002b). Evaluating the effectiveness of ensembles of decision trees in disambiguating SENSEVAL lexical samples. In Proceedings of the ACL-02 workshop on word sense disambiguation: Recent successes and future directions (pp. 81–87). · DOI
  39. Piatetsky-Shapiro, G. (1999). Discovery, analysis and presentation of strong rules. In G. Piatetsky-Shapiro & W. J. Frawley (Eds.), Knowledge discovery in databases (pp. 229–248). Menlo Park, CA: AAAI Press.
  40. Poesio, M., & Artstein, R. (2005). The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of the workshop on frontiers in corpus annotation II: Pie in the sky (pp. 76–83).
  41. Pradhan, S., Loper, E., Dligach, D., & Palmer, M. (2007). SemEval-2007 Task-17: English lexical sample, SRL and all words. In Proceedings of 4th international workshop on semantic evaluations (SemEval-2007), Prague, Czech Republic (pp. 87–92). · DOI
  42. Raykar, V. C., Yu, S., Zhao, L. H., Jerebko, A., Florin, C., Valadez, G. H., Bogoni, L., & Moy, L. (2009). Supervised learning from multiple experts: whom to trust when everyone lies a bit. In Proceedings of the 26th annual international conference on machine learning (ICML 09), New York, NY (pp. 889–896). · DOI
  43. Raykar, V. C., Yu, S., Zhao, L. H., Valadez, G. H., Florin, C., Bogoni, L., & Moy, L. (2010). Learning from crowds. Journal of Machine Learning Research, 11, 1297–1322.
  44. Resnik, P., & Yarowsky, D. (1999). Distinguishing systems and distinguishing senses: New evaluation methods for word sense disambiguation. Natural Language Engineering, 5(2), 113–133. · DOI
  45. Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly, 17, 321–325. · DOI · Paper page
  46. Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008). Get another label? improving data quality and data mining using multiple noisy labelers. In Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, ACM, New York, NY, USA, KDD ’08 (pp. 614–622). · DOI
  47. Snow, R., Jurafsky, D., & Ng, A. Y. (2007). Learning to merge word senses. In Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning, Prague (pp. 1005–1014).
  48. Snow, R., O’Connor, B., Jurafsky, D., & Ng, A. Y. (2008). Cheap and fast - but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP), Honolulu (pp. 254–263). · DOI · Paper page
  49. Sorokin, A., & Forsyth, D. (2008). Utility data annotation with Amazon Mechanical Turk. In Computer ision and Pattern Recognition Workshops (CVPRW 08), First IEEE workshop on internet vision, pp. 1–8. · DOI
  50. Véronis, J. (1998). A study of polysemy judgements and inter-annotator agreement. In SENSEVAL Workshop, Sussex.
  51. Whitehill, J., Ruvolo, P., Wu, T. fan, Bergsma, J., & Movellan, J. (2000). Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, & A. Culotta (Eds.), Advances in Neural Information Processing Systems 22 (pp. 2035–2043). Cambridge: MIT Press.
  52. Yan, Y., Rosales, R., Fung, G., Schmidt, M., Hermosillo, G., Bogoni, L., Moy, L. G., & Dy, J. (2010). Modeling annotator expertise: Learning when everybody knows a bit of something. In Proceedings of the 13th international conference on artificial intelligence and statistics (AISTATS) (pp. 932–939).

Cited by

Sorted by citations

Metadata

Authors
Passonneau, R. J., Bhardwaj, V., Salleb-Aouissi, A., & Ide, N.
Affiliations
  • Columbia University (US)
  • Vassar College (US)
Corresponding author
Passonneau, Rebecca J.
ORCID
Venue
Language Resources and Evaluation
Publisher
Springer Science and Business Media LLC
Published
2012-05-15
Volume
46
Issue
2
Pages
219-252
ISSN
  • 1574-020X (ISSN-L)
  • 1574-0218
DOI
10.1007/s10579-012-9188-x
OpenAlex
W2058959415
Cited by
34
References
0
Citations by year
2025: 2 · 2024: 1 · 2023: 4 · 2022: 3 · 2021: 8 · 2019: 2 · 2018: 1 · 2017: 3 · 2015: 1 · 2014: 4 · 2013: 2 · 2012: 3
FWCI
3.15
Type
journal article
Language
en
Open access
closed
Licence
www.springer.com/tdm (tdm)
Locations
Language Resources and Evaluation (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
Keywords
word sense disambiguation, annotation agreement, ground truth annotation, polysemy, annotation reliability, sense inventory

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review