The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
SemEval-2018 Task 3: Irony Detection in English Tweets
Open accessAbstract
This paper presents the first shared task on irony detection: given a tweet, automatic natural language processing systems should determine whether the tweet is ironic (Task A) and which type of irony (if any) is expressed (Task B).The ironic tweets were collected using irony-related hashtags (i.e.#irony, #sarcasm, #not) and were subsequently manually annotated to minimise the amount of noise in the corpus.Prior to distributing the data, hashtags that were used to collect the tweets were removed from the corpus.For both tasks, a training corpus of 3,834 tweets was provided, as well as a test set containing 784 tweets.Our shared tasks received submissions from 43 teams for the binary classification Task A and from 31 teams for the multiclass Task B. The highest classification scores obtained for both subtasks are respectively F 1 = 0.71 and F 1 = 0.51 and demonstrate that fine-grained irony classification is much more challenging than binary irony detection.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · cited 1×
“NLPRL-IITBHU also used the same architecture, but given the data skew for Task B, they used SMOTE (Chawla et al., 2002) as an oversampling technique to make sure each irony class was equally represented in the training corpus, which lead to an F 1 score increase of 5 points.”
MethodMeasuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“As metric, we used Fleiss’ Kappa (Fleiss, 1971), a widespread statistical measure in the field of computational linguistics for assessing annotator agreement on categorical ratings (Carletta, 1996).”
MethodMining and summarizing customer reviews
Hu, M., & Liu, B. (2004) · Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1014052.1014073 · Cited by 7,793
VADER: a parsimonious rule-based model for sentiment analysis of social media text
Hutto, C., & Gilbert, E. (2014) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v8i1.14550 · Cited by 6,087
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2068 · Cited by 4,545
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2016) · arXiv · doi:10.48550/arxiv.1607.01759 · Cited by 4,542
Sentiment Analysis and Opinion Mining
Zhang, L., & Liu, B. (2017) · Encyclopedia of Machine Learning and Data Mining · doi:10.1007/978-1-4899-7687-1_907 · Cited by 3,260
(1992) · doi:10.5555/176313.176316 · Cited by 2,892
SIGNATURE VERIFICATION USING A “SIAMESE” TIME DELAY NEURAL NETWORK
BROMLEY, J., BENTZ, J. W., BOTTOU, L., GUYON, I., LECUN, Y., MOORE, C., SÄCKINGER, E., & SHAH, R. (1994) · Series in Machine Perception and Artificial Intelligence · doi:10.1142/9789812797926_0003 · Cited by 2,350
Assessing Agreement on Classification Tasks: The Kappa Statistic
Carletta, J. (1996) · arXiv · doi:10.48550/arxiv.cmp-lg/9602004 · Cited by 2,088 · cited 2×
“As metric, we used Fleiss’ Kappa (Fleiss, 1971), a widespread statistical measure in the field of computational linguistics for assessing annotator agreement on categorical ratings (Carletta, 1996).”
MethodSignature Verification Using A "Siamese" Time Delay Neural Network
BROMLEY, J., BENTZ, J. W., BOTTOU, L., GUYON, I., LECUN, Y., MOORE, C., SÄCKINGER, E., & SHAH, R. (1993) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001493000339 · Cited by 2,077
SemEval-2016 Task 4: Sentiment Analysis in Twitter
Nakov, P., Ritter, A., Rosenthal, S., Sebastiani, F., & Stoyanov, V. (2016) · Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016) · doi:10.18653/v1/s16-1001 · Cited by 966
SemEval-2017 Task 4: Sentiment Analysis in Twitter
Rosenthal, S., Farra, N., & Nakov, P. (2017) · Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) · doi:10.18653/v1/s17-2088 · Cited by 870
A new ANEW: Evaluation of a word list for sentiment analysis in microblogs
Nielsen, F. Å. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.2903 · Cited by 712
Felbo, B., Mislove, A. E., Søgaard, A., Rahwan, I., & Lehmann, S. (2017) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 712
Identifying Sarcasm in Twitter: A Closer Look
González‐Ibáñez, R., Muresan, S., & Wacholder, N. (2011) · Cited by 627
SemEval-2014 Task 9: Sentiment Analysis in Twitter
Rosenthal, S., Ritter, A., Nakov, P., & Stoyanov, V. (2014) · Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014) · doi:10.3115/v1/s14-2009 · Cited by 617
Sarcasm as Contrast between a Positive Sentiment and Negative Situation
Riloff, E., Qadir, A., Surve, P., De Silva, L., Gilbert, N., & Huang, R. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1066 · Cited by 587
Sentiment Analysis and Opinion Mining
Zhang, L., & Liu, B. (2023) · Encyclopedia of Machine Learning and Data Science · doi:10.1007/978-1-4899-7502-7_907-2 · Cited by 518
A multidimensional approach for detecting irony in Twitter
Reyes, A., Rosso, P., & Veale, T. (2013) · Language Resources and Evaluation · doi:10.1007/s10579-012-9196-x · Cited by 454
Semi-supervised recognition of sarcastic sentences in Twitter and Amazon
Davidov, D., Tsur, O., & Rappoport, A. (2010) · Cited by 424
Automatic Sarcasm Detection: A Survey
Joshi, A., Bhattacharyya, P., & Carman, M. J. (2018) · ACM Computing Surveys · doi:10.1145/3124420 · Cited by 411
Contextualized Sarcasm Detection on Twitter
Bamman, D., & Smith, N. (2021) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v9i1.14655 · Cited by 352
Who cares about Sarcastic Tweets? Investigating the Impact of Sarcasm on Sentiment Analysis.
Maynard, D., & Greenwood, M. (2014) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2ios4bdd5xas · Cited by 349
SenticNet 4: A Semantic Resource for Sentiment Analysis Based on Conceptual Primitives
Cambria, E., Poria, S., Bajpai, R., & Schuller, B. W. (2016) · OPUS (Augsburg University) · Cited by 347
Fracking Sarcasm using Neural Network
Ghosh, A., & Veale, D. T. (2016) · Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis · doi:10.18653/v1/w16-0425 · Cited by 290
Modelling Context with User Embeddings for Sarcasm Detection in Social Media
Amir, S., Wallace, B. C., Lyu, H., Carvalho, P., & Silva, M. J. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1017 · Cited by 272
Mohammad, S., & Bravo-Marquez, F. (2017) · Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017) · doi:10.18653/v1/s17-1007 · Cited by 220
Sarcasm Detection on Czech and English Twitter
Ptáċek, T., Habernal, I., & Hong, J. S. (2014) · Research Portal (Queen's University Belfast) · Cited by 170
Irony Detection in Twitter: The Role of Affective Content
Farías, D. I. H., Patti, V., & Rosso, P. (2016) · ACM Transactions on Internet Technology · doi:10.1145/2930663 · Cited by 154
The bicoherence theory of situational irony
Shelley, C. (2001) · Cognitive Science · doi:10.1207/s15516709cog2505_7 · Cited by 113
Barbieri, F., & Saggion, H. (2014) · Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop (EACL SRW) · doi:10.3115/v1/e14-3007 · Cited by 112
Wu, C., Wu, F., Wu, S., Liu, J., Yuan, Z., & Huang, Y. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1006 · Cited by 112
Automatic Sarcasm Detection: A SurveyPreprint
Joshi, A., Bhattacharyya, P., & Carman, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.03426 · Cited by 108
SemEval-2017 Task 6: #HashtagWars: Learning a Sense of Humor
Potash, P., Romanov, A., & Rumshisky, A. (2017) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s17-2004 · Cited by 69
Harnessing Cognitive Features for Sarcasm Detection
Mishra, A., Kanojia, D., Nagar, S., Dey, K., & Bhattacharyya, P. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1104 · Cited by 64
Monday mornings are my fave :) #not Exploring the Automatic Recognition of Irony in English tweets
Hee, C. V., Lefever, E., & Hoste, V. (2016) · Ghent University Academic Bibliography (Ghent University) · Cited by 50
Self-citationWLV at SemEval-2018 Task 3: Dissecting Tweets in Search of Irony
Rohanian, O., Taslimipoor, S., Evans, R., & Mitkov, R. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1090 · Cited by 32
Gupta, R. K., & Yang, Y. (2017) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s17-2103 · Cited by 31
Vu, T., Nguyen, D. Q., Vu, X.-S., Nguyen, D. Q., Catt, M., & Trenell, M. I. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1085 · Cited by 27
Rangwani, H., Kulshreshtha, D., & Singh, A. K. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1104 · Cited by 23
IronyMagnet at SemEval-2018 Task 3: A Siamese network for Irony detection in Social media
Ghosh, A., & Veale, T. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1093 · Cited by 23
(2012) · Research Explorer (The University of Manchester)
Cited by
BERTweet: A pre-trained language model for English Tweets
Nguyen, D. Q., Vu, T., & Tuan Nguyen, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.2 · Cited by 843
TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification
Barbieri, F., Camacho-Collados, J., Espinosa Anke, L., & Neves, L. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.148 · Cited by 674
A transformer-based approach to irony and sarcasm detection
Potamias, R. A., Siolas, G., & Stafylopatis, A. G. (2020) · Neural Computing and Applications · doi:10.1007/s00521-020-05102-3 · Cited by 278
TimeLMs: Diachronic Language Models from Twitter
Loureiro, D., Barbieri, F., Neves, L., Espinosa Anke, L., & Camacho-collados, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.25 · Cited by 263
Li, Z., Zhu, H., Lu, Z., & Yin, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.647 · Cited by 139
Metadata
- Authors
- Van Hee, C., Lefever, E., & Hoste, V.
- Venue
- Proceedings of The 12th International Workshop on Semantic Evaluation
- Published
- 2018
- Pages
- 39-50
- OpenAlex
- W2807333695
- Cited by
- 292
- References
- 46
- FWCI
- 14.88
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.