Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 2,634

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.

CROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON

Open access
Mohammad, Saif M., Turney, Peter D.·2013·Computational Intelligence
DOI 10.1111/j.1467-8640.2012.00460.x
Open this paper in CoChat — free account

Abstract

Even though considerable attention has been given to the polarity of words (positive and negative) and the creation of large polarity lexicons, research in emotion analysis has had to rely on limited and small emotion lexicons. In this paper, we show how the combined strength and wisdom of the crowds can be used to generate a large, high‐quality, word–emotion and word–polarity association lexicon quickly and inexpensively. We enumerate the challenges in emotion annotation in a crowdsourcing scenario and propose solutions to address them. Most notably, in addition to questions about emotions associated with terms, we show how the inclusion of a word choice question can discourage malicious data entry, help to identify instances where the annotator may not be familiar with the target term (allowing us to reject such annotations), and help to obtain annotations at sense level (rather than at word level). We conducted experiments on how to formulate the emotion‐annotation questions, and show that asking if a term is associated with an emotion leads to markedly higher interannotator agreement than that obtained by asking if a term evokes an emotion.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1111/j.1467-8640.2012.00460.x

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 11 references

Research works printed but not linked: 91

  1. 10.1145/1367497.1367587 · DOI · Paper page
  2. 10.3115/1220575.1220648 · DOI · Paper page
  3. 10.1007/978-3-540-74628-7_27 · DOI · Paper page
  4. Ross W. D., 1913, The Works of Aristotle, 805
  5. Bellegarda J., 2010, Proceedings of the NAACL‐HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text
  6. J. Bernard, 1986, The Macquarie Thesaurus. Macquarie Library
  7. Boucouvalas A. C., 2002, Emerging Communication: Studies on New Technologies and Practices in Communication, 305
  8. Bougie J. R. G., 2003, Open access publications from Tilburg University
  9. Bradley M., 1999, Technical Report, C‐1, The Center for Research in Psychophysiology
  10. Brants T., 2006, Web 1T 5‐Gram Version 1, Linguistic Data Consortium
  11. Breazeal C., 2004, Who Needs Emotions
  12. 10.1177/001316448104100307 · DOI · Paper page
  13. 10.3115/1699510.1699548 · DOI · Paper page
  14. 10.1177/001316446002000104 · DOI · Paper page
  15. Cohen S. H, 2003, Maximum difference scaling: Improved measures of importance and preference for segmentation, Sawtooth Software Conference Proceedings, Sawtooth Software, Inc., 61
  16. 10.1016/S0022-4359(00)00029-4 · DOI · Paper page
  17. 10.1037/10001-000 · DOI · Paper page
  18. Davenport T. H., 2001, How do they know their customers so well?, MIT Sloan Management Review, 42, 63
  19. 10.1108/09564230210425331 · DOI
  20. 10.1016/S1067-5671(96)05054-8 · DOI
  21. Ekman P., 2005, Emotion in the Human Face
  22. Ekman P., 2003, Malor Books
  23. Elliott C., 1992, Ph.D. Thesis, Institute for the Learning Sciences
  24. 10.1086/209086 · DOI · Paper page
  25. 10.1007/11846406_47 · DOI
  26. Genereux M., 2006, Distinguishing affective states in weblogs, AAAI-2006 Spring Symposium on Computational Approaches to Analysing Weblogs, Stanford, CA, 27
  27. Goyal A., 2010, Proceedings of the NAACL‐HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text
  28. 10.1007/s10071-008-0199-3 · DOI · Paper page
  29. 10.1016/j.jretai.2004.04.002 · DOI · Paper page
  30. 10.1109/IROS.2006.282327 · DOI
  31. Holsti O. R.1969.Content analysis for the social sciences and humanities.Addison‐Wesley: Reading MA .
  32. Holzman L. E., 2003, Technical report
  33. Howe J., 2006, Crowdsourcing: Tracking the Rise of the Amateur
  34. 10.1093/mind/os-IX.34.188 · DOI · Paper page
  35. John D., 2006, Proceedings of the 24th IASTED International Conference on Internet and Multimedia Systems and Applications, 183
  36. 10.1007/11555261_81 · doi:10.1007/11555261_81
  37. 10.1145/1835449.1835618 · doi:10.1145/1835449.1835618
  38. Kŏvecses Z, 2003, Culture, and Body in Human Feeling Studies in Emotion and Social Interaction
  39. Lazarus R. S, 2000, Handbook of Cognition and Emotion, 1
  40. Litman D. J., 2004, Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, ACL ’04, USA
  41. 10.1145/604045.604067 · DOI · Paper page
  42. Louviere J. J., 1991, Technical report
  43. Louviere J. J., 1992, Determining the appropriate response to evidence of public concern: The case of food safety, Journal of Public Policy and Marketing, 11, 12, 10.1177/074391569201100202 · DOI · Paper page
  44. Ma C., 2005, First International Conference on Affective Computing and Intelligent Interaction ACII‐2005, 622
  45. Masson J. M., 1996, When Elephants Weep: The Emotional Lives of Animals
  46. Matykiewicz P., 2009, Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing, BioNLP ’09, USA, 179
  47. 10.1016/0167-4870(93)90001-2 · DOI · Paper page
  48. Mihalcea R., 2006, AAAI‐2006 Spring Symposium on Computational Approaches to Analysing Weblogs, 139
  49. Mohammad S. M., 2011, Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies
  50. Mohammad S. M., 2011, Proceedings of the ACL 2011 Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities LaTeCH
  51. Mohammad S. M., 2010, Proceedings of the NAACL‐HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text
  52. Mohammad S. M., 2011, Proceedings of the ACL 2011 Workshop on Computational Approaches to Subjectivity and Sentiment Analysis WASSA
  53. 10.3115/1699571.1699591 · DOI
  54. Neviarouskaya A., 2009, Proceedings of the Third International Conference on Weblogs and Social Media ICWSM‐09, 278
  55. 10.1017/S1351324910000239 · DOI
  56. 10.1111/j.1749-6632.1956.tb49644.x · DOI · Paper page
  57. Oliver R. L.1997.Satisfaction a behavioral perspective on the consumer.McGraw‐Hill: New York .
  58. 10.1037/0033-295X.97.3.315 · DOI · Paper page
  59. 10.1037/h0047078 · DOI
  60. 10.1561/1500000011 · DOI · Paper page
  61. Parrot W., 2001, Emotions in Social Psychology
  62. Pearl L., 2010, Proceedings of the NAACL‐HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text
  63. 10.3115/1572306.1572327 · DOI
  64. Plutchik R., 1962, The Emotions
  65. 10.1016/B978-0-12-558701-3.50007-7 · DOI · Paper page
  66. 10.1007/BF00991576 · DOI
  67. Plutchik R., 1994, The Psychology and Biology of Emotion
  68. 10.1007/11788713_5 · DOI
  69. 10.1162/pres.15.4.381 · DOI · Paper page
  70. Read J., 2004, Ph.D. Thesis, Department of Informatics
  71. Reichheld F. F., 2000, E‐loyalty: Your secret weapon on the web, Harvard Business Review, 78, 105
  72. Rentoumi V., 2009, Proceedings of the International Conference RANLP‐2009, Borovets, Bulgaria, 370
  73. 10.1007/BF02722168 · DOI · Paper page
  74. Richins M. L., 1984, Word of mouth communication as negative information, Advances in Consumer Research, 11, 697
  75. 10.1037/0033-2909.115.1.102 · DOI · Paper page
  76. 10.1177/053901882021004004 · DOI · Paper page
  77. Scherer K. R., 1984, Emotion as a multicomponent process: A model and some cross‐cultural data, Review of Personality and Social Psychology, 5, 37
  78. 10.1086/266577 · DOI · Paper page
  79. 10.1016/S0963-8687(02)00022-7 · DOI · Paper page
  80. 10.2307/1251688 · DOI
  81. 10.3115/1613715.1613751 · DOI · Paper page
  82. Steunebrink B. R, 2010, Ph.D. Thesis
  83. Stewart K. J., 2003, Trust transfer on the World Wide Web. Organization Science, 5
  84. Stone P., 1966, The General Inquirer: A Computer Approach to Content Analysis
  85. Strapparava C., 2004, Proceedings of the 4th International Conference on Language Resources and Evaluation LREC‐2004, 1083
  86. 10.1109/91.940962 · DOI · Paper page
  87. 10.1145/944012.944013 · DOI · Paper page
  88. Velásquez J. D., 1997, Proceedings of the Fourteenth National Conference on Artificial Intelligence and Ninth Conference on Innovative Applications of Artificial Intelligence, AAAI’97/IAAI’97, 10
  89. Wiebe J. M., 1994, Tracking point of view in narrative, Computational Linguistics, 20, 233
  90. 10.1037/0003-066X.39.2.117 · doi:10.1037/0003-066x.39.2.117 · Paper page
  91. Zhe X., 2002, Text‐to‐emotion engine for real time Internet communication, International Symposium on CSNDSP, 164

Cited by

See all 2,634

Metadata

Authors
Mohammad, S. M., & Turney, P. D.
Venue
Computational Intelligence
Publisher
Wiley
Published
2012-09-04
Volume
29
Issue
3
Pages
436-465
ISSN
  • 0824-7935 (ISSN-L)
  • 1467-8640
DOI
10.1111/j.1467-8640.2012.00460.x
OpenAlex
W2040467972
Cited by
2,634
References
11
FWCI
39.58
Type
journal article
Language
en
Open access
hybrid
Licence
onlinelibrary.wiley.com/termsAndConditions#vor (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: CROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON. https://app.cochat.ai/papers/10.1111/j.1467-8640.2012.00460.x
The paper
Mohammad, S. M., & Turney, P. D. (2013). CROWDSOURCING A WORD–EMOTION ASSOCIATION LEXICON. Computational Intelligence. https://doi.org/10.1111/j.1467-8640.2012.00460.x

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.