CoChat
Sign Up
Cited by 193Misinformation and Its ImpactsSentiment Analysis and Opinion MiningHate Speech and Cyberbullying Detection

SemEval-2019 Task 4: Hyperpartisan News Detection

Open access
Kiesel, Johannes, Mestre, Maria, Shukla, Rishabh, Vincent, Emmanuel, Adineh, Payam, Corney, David, Stein, Benno, Potthast, Martin·2019·Proceedings of the 13th International Workshop on Semantic Evaluation
DOI 10.18653/v1/s19-2145
Open this paper in CoChat — free accountOpen PDF

Abstract

Hyperpartisan news is news that takes an extreme left-wing or right-wing standpoint.If one is able to reliably compute this meta information, news articles may be automatically tagged, this way encouraging or discouraging readers to consume the text.It is an open question how successfully hyperpartisan news detection can be automated, and the goal of this SemEval task was to shed light on the state of the art.We developed new resources for this purpose, including a manually labeled dataset with 1,273 articles, and a second dataset with 754,000 articles, labeled via distant supervision.The interest of the research community in our task exceeded all our expectations: The datasets were downloaded about 1,000 times, 322 teams registered, of which 184 configured a virtual machine on our shared task cloud service TIRA, of which in turn 42 teams submitted a valid run.The best team achieved an accuracy of 0.822 on a balanced sample (yes : no hyperpartisan) drawn from the manually tagged corpus; an ensemble of the submitted systems increased the accuracy by 0.048.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/s19-2145

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In contrast to our multi-dimensional approach that examines the ideology of the issue being discussed instead of the author stance, much of the recent work in computational linguistics has been dedicated to the latter while collapsing ideology to one dimension.

    “In contrast to our multi-dimensional approach that examines the ideology of the issue being discussed instead of the author stance, much of the recent work in computational linguistics has been dedicated to the latter (detection of ideological bias in news media) while collapsing ideology to one dimension”

    From Political Ideology and Polarization: A Multi-dimensional Approach (2022) · cites this paper as Kiesel et al., 2019

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 47

  1. Nazanin Afsarmanesh, Jussi Karlgren, Peter Sumbler, and Nina Viereckel. 2019. Team Harry Friberg at SemEval-2019 Task 4: Identifying Hyperpartisan News through Editorially Defined Metatopics. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  2. Rodrigo Agerri. 2019. Doris Martin at SemEval-2019 Task 4: Hyperpartisan News Detection with Generic Semi-supervised Featuresl. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  3. Amal Alabdulkarim and Tariq Alhindi. 2019. Spider-Jerusalem at SemEval-2019 Task 4: Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  4. Carla Perez Almendros, Luis Espinosa Anke, and Steven Schockaert. 2019. Cardiff University at SemEval-2019 Task 4: Linguistic Features for Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  5. Evan Amason, Jake Palanker, Mary Clare Shen, and Julie Medero. 2019. Harvey Mudd College at SemEval-2019 Task 4: The D.X. Beaumont Hyperpartisan News Detector. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  6. Team Kermit-the-frog at SemEval-2019 Task 4: Bias Detection Through Sentiment Analysis and Simple Linguistic Features. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  7. Yves Bestgen. 2019. Tintin at SemEval-2019 Task 4: Detecting Hyperpartisan News Article with only Simple Tokens. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  8. Shweta Bhatt, Sagar Joglekar, Shehar Bano, and Nishanth Sastry. 2018. Illuminating the ecosystem of partisan websites. In Proceedings of the 27th International Conference on World Wide Web Companion, WWW '18 Companion. International World Wide Web Conferences Steering Committee.
  9. Nikhil Chakravartula, Vijayasaradhi Indurthi, and Bakhtiyar Syed. 2019. Fermi at SemEval-2019 Task 4: The sarah-jane-smith Hyperpartisan News Detector. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  10. Celena Chen, Celine Park, Jason Dwyer, and Julie Medero. 2019. Harvey Mudd College at SemEval-2019 Task 4: The Carl Kolchak Hyperpartisan News Detector. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  11. Rebekah Cramerus and Tatjana Scheffler. 2019. Team Kit Kittredge at SemEval-2019 Task 4. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  12. André Cruz, Gil Rocha, Rui Sousa-Silva, and Henrique Lopes Cardoso. 2019. Team Fernando-Pessa at SemEval-2019 Task 4: Back to Basics in Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  13. Mehdi Drissi, Pedro Sandoval Segura, Vivaswat Ojha, and Julie Medero. 2019. Harvey Mudd College at SemEval-2019 Task 4: The Clint Buchanan Hyperpartisan News Detector. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  14. Michael Färber, Agon Qurdina, and Lule Ahmedi. 2019. Team Peter Brinkmann at SemEval-2019 Task 4: Detecting Biased News Articles Using Convolutional Neural Networks. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  15. Eibe Frank, Mark A. Hall, and Ian H. Witten. 2016. Data Mining: Practical Machine Learning Tools and Techniques, 4th edition, chapter The WEKA Workbench. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
  16. Viresh Gupta, Baani Leen Kaur Jolly, Ramneek Kaur, and Tanmoy Chakraborty. 2019. Clark Kent at SemEval-2019 Task 4: Stylometric Insights into Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  17. Matthias Hagen, Martin Potthast, Michel Büchner, and Benno Stein. 2015. Webis: An Ensemble for Twitter Sentiment Detection. In 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 582-589. Association for Computational Linguistics.
  18. Kazuaki Hanawa, Shota Sasaki, Hiroki Ouchi, Jun Suzuki, and Kentaro Inui. 2019. The Sally Smedley Hyperpartisan News Detector at SemEval-2019 Task 4. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  19. Jeremy Howard and Sebastian Ruder. 2018. Universal Language Model Fine-tuning for Text Classification. CoRR, abs/1801.06146. · arXiv
  20. Dick-Preston and Morbo at SemEval-2019 Task 4: Transfer Learning for Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  21. Ye Jiang, Johann Petrak, Xingyi Song, Kalina Bontcheva, and Diana Maynard. 2019. Team Bertha von Suttner at SemEval-2019 Task 4: Hyperpartisan News Detection using ELMo Sentence Representation Convolutional Network. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  22. Youngjun Joo and Inchon Hwang. 2019. Steve Martin at SemEval-2019 Task 4: Ensemble Learning Model for Detecting Hyperpartisan News. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  23. Johannes Kiesel, Florian Kneist, Milad Alshomary, Benno Stein, Matthias Hagen, and Martin Potthast. 2018. Reproducible Web Corpora: Interactive Archiving with Automatic Quality Assessment. Journal of Data and Information Quality (JDIQ), 10(4):17:1-17:25. · DOI
  24. Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classification. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). · DOI · Paper page
  25. Jürgen Knauth. 2019. Orwellian-times at SemEval-2019 Task 4: A Stylistic and Content-based Classifier. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  26. Nayeon Lee, Zihan Liu, and Pascale Fung. 2019. Team yeon-zi at SemEval-2019 Task 4: Hyperpartisan News Detection by De-noising Weakly-labeled Data. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  27. Jose G. Moreno, Yoann Pitarch, Karen Pinel-Sauvagnat, and Gilles Hubert. 2019. Rouletabille at SemEval-2019 Task 4: Neural Network Baseline for Identification of Hyperpartisan Publishers. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  28. Osman Mutlu, Ozan Arkan Can, and Erenay Dayanik. 2019. Team Howard Beale at SemEval-2019 Task 4: Hyperpartisan News Detection with BERT. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  29. Duc-Vu Nguyen, Thin Dang, and Ngan Nguyen. 2019. NLP@UIT at SemEval-2019 Task 4: The Paparazzo Hyperpartisan News Detector. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  30. Zhiyuan Ning, Yuanzhen Lin, and Ruichao Zhong. 2019. Team Peter-Parker at SemEval-2019 Task 4: BERT-Based Method in Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  31. Niko Palić, Juraj Vladika, Dominik Cubelić, Ivan Lovrencic, and Jan Snajder. 2019. TakeLab at SemEval-2019 Task 4: Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  32. Olga Papadopoulou, Giorgos Kordopatis-Zilos, Markos Zampoglou, Symeon Papadopoulos, and Yiannis Kompatsiaris. 2019. Brenda Starr at SemEval-2019 Task 4: Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  33. Martin Potthast, Tim Gollub, Matthias Hagen, and Benno Stein. 2018a. The Clickbait Challenge 2017: Towards a Regression Model for Clickbait Strength. CoRR, abs/1812.10847. · arXiv
  34. Martin Potthast, Tim Gollub, Kristof Komlossy, Sebastian Schuster, Matti Wiegmann, Erika Patricia Garces Fernandez, Matthias Hagen, and Benno Stein. 2018b. Crowdsourcing a Large Corpus of Clickbait on Twitter. In 27th International Conference on Computational Linguistics (COLING 2018), pages 1498-1507. The COLING 2018 Organizing Committee.
  35. Martin Potthast, Tim Gollub, Matti Wiegmann, and Benno Stein. 2019. TIRA Integrated Research Architecture. In Nicola Ferro and Carol Peters, editors, Information Retrieval Evaluation in a Changing World -Lessons Learned from 20 Years of CLEF. Springer.
  36. Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2018c. A Stylometric Inquiry into Hyperpartisan and Fake News. In 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018), pages 231-240. Association for Computational Linguistics.
  37. Martin Potthast, Sebastian Köpsel, Benno Stein, and Matthias Hagen. 2016. Clickbait Detection. In Advances in Information Retrieval. 38th European Conference on IR Research (ECIR 2016), volume 9626 of Lecture Notes in Computer Science, pages 810-817, Berlin Heidelberg New York. Springer. · DOI
  38. Abdelrhman Saleh, Ramy Baly, Alberto Barrón-Cedeño, Giovanni Da San Martino, Mitra Mohtarami, Preslav Nakov, and James Glass. 2019. Team QCRI-MIT at SemEval-2019 Task 4: Propaganda Analysis Meets Hyperpartisan News Detection. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  39. Saptarshi Sengupta and Ted Pedersen. 2019. Duluth at SemEval-2019 Task 4: The Pioquinto Manterola Hyperpartisan News Detector. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  40. Daniel Shaprin, Giovanni Da San Martino, Alberto Barrón-Cedeño, and Preslav Nakov. 2019. Team Jack Ryder at SemEval-2019 Task 4: Using BERT Representations for Detecting Hyperpartisan News. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  41. Vertika Srivastava, Ankita Gupta, Divya Prakash, Sudeep Sahoo, Rohit R. R., and Yeon Hyang Kim. 2019. Vernon-fenwick at SemEval-2019 Task 4: Hyperpartisan News Detection using Lexical and Semantic Features. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  42. Bozhidar Stevanoski and Sonja Gievska. 2019. Team Ned Leeds at SemEval-2019 Task 4: Exploring Language Indicators of Hyperpartisan Reporting. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  43. Emmanuel Vincent and Maria Mestre. 2018. Crowdsourced Measure of News Articles Bias: Assessing Contributors' Reliability. In Proceedings of the 1st Workshop on Subjectivity, Ambiguity and Disagreement (SAD) in Crowdsourcing, pages 1-10.
  44. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. 2016. Hierarchical Attention Networks for Document Classification. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480-1489.
  45. Chia-Lun Yeh, Babak Loni, and Anne Schuth. 2019. Tom Jumbo-Grumbo at SemEval-2019 Task 4: Hyperpartisan News Detection with GloVe vectors and SVM. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  46. Albin Zehe, Lena Hettinger, Stefan Ernst, Christian Hauptmann, and Andreas Hotho. 2019. Team Xenophilius Lovegood at SemEval-2019 Task 4: Hyperpartisanship Classification using Convolutional Neural Networks. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.
  47. Chiyu Zhang, Arun Rajendran, and Muhammad Abdul-Mageed. 2019. UBC-NLP at SemEval-2019 Task 4: Hyperpartisan News Detection With Attention-Based Bi-LSTMs. In Proceedings of The 13th International Workshop on Semantic Evaluation. Association for Computational Linguistics.

Cited by

Sorted by citations

Authors
Kiesel, J., Mestre, M., Shukla, R., Vincent, E., Adineh, P., Corney, D., Stein, B., & Potthast, M.
Affiliations
  • Bauhaus-Universität Weimar (DE)
  • Leipzig University (DE)
ORCID
Venue
Proceedings of the 13th International Workshop on Semantic Evaluation
Publisher
Association for Computational Linguistics
Published
2019
Pages
829-839
DOI
10.18653/v1/s19-2145
OpenAlex
W2955041501
Cited by
193
References
2
Citations by year
2026: 8 · 2025: 13 · 2024: 16 · 2023: 39 · 2022: 32 · 2021: 36 · 2020: 18 · 2019: 31
FWCI
83.39
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
International Workshop on Semantic Evaluation (SemEval) (publishedVersion, cc-by)
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
  • Physical Sciences › Computer Science › Artificial Intelligence › Hate Speech and Cyberbullying Detection
Keywords
political bias detection, news articles, distant supervision, shared task, ensemble systems

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review