CoChat

A Structured Response to Misinformation: Defining and Annotating Credibility Indicators in News Articles

Open access
Zhang, Amy X., Robbins, Martin, Bice, Ed, Hawke, Sandro, Karger, David, Mina, An Xiao, Ranganathan, Aditya, Metz, Sarah Emlen, Appling, Scott, Sehat, Connie Moon, Gilmore, Norman, Adams, Nick B., Vincent, Emmanuel, Lee, Jennifer · 2018 · Companion of the The Web Conference 2018 on The Web Conference 2018 - WWW '18
Cited by 120

The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.

Abstract

The proliferation of misinformation in online news and its amplification by platforms are a growing concern, leading to numerous efforts to improve the detection of and response to misinformation. Given the variety of approaches, collective agreement on the indicators that signify credible content could allow for greater collaboration and data-sharing across initiatives. In this paper, we present an initial set of indicators for article credibility defined by a diverse coalition of experts. These indicators originate from both within an article's text as well as from external sources or article metadata. As a proof-of-concept, we present a dataset of 40 articles of varying credibility annotated with our indicators by 6 trained annotators using specialized platforms. We discuss future steps including expanding annotation, broadening the set of indicators, and considering their use by platforms and the public, towards the development of interoperable standards for content credibility.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3184558.3188731

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 10 references

Research works printed but not linked: 32

  1. Ahmer Arif, John J Robinson, Stephanie A Stanek, Elodie S Fichet, Paul Townsend, Zena Worku, and Kate Starbird. 2017. A Closer Look at the Self-Correcting Crowd: Examining Corrections in Online Rumors. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 155--168. · DOI · Paper page
  2. Ahmer Arif, Kelley Shanahan, Fang-Ju Chou, Yoanna Dosouto, Kate Starbird, and Emma S Spiro. 2016. How information snowballs: Exploring the role of exposure in online rumor propagation. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 466--477.
  3. Leticia Bode and Emily K Vraga. 2015. In related news, that was wrong: The correction of misinformation through related stories functionality in social media. Journal of Communication 65, 4 (2015), 619--638. · DOI · Paper page
  4. David B. Buller and Judee K. Burgoon. 1996. Interpersonal Deception Theory. Communication Theory 6, 3 (1996), 203--242. https://doi.org/10.1111/j.1468- 2885.1996.tb00127.x · doi:10.1111/j.1468-
  5. Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. 2011. Information credibility on twitter. In Proceedings of the 20th international conference on World wide web. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 675--684. · DOI · Paper page
  6. Michael A. Caulfield. 2017. Go Upstream to the Find the Source. (Jan 2017). https://webliteracy.pressbooks.com/chapter/go-upstream-to-find-the-source/
  7. Yimin Chen, Niall J Conroy, and Victoria L Rubin. 2015. Misleading online content: Recognizing clickbait as false news. In Proceedings of the 2015 ACM on Workshop on Multimodal Deception Detection. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 15--19.
  8. Andrew J Flanagin and Miriam J Metzger. 2000. Perceptions of Internet information credibility. Journalism & Mass Communication Quarterly 77, 3 (2000), 515--540. · DOI · Paper page
  9. William B. Frakes. 1986. Information and misinformation: An investigation of the notions of information, misinformation, informing, and misinforming. Journal of the American Society for Information Science 37, 1 (1986), 48--49. https: //doi.org/10.1002/(SICI)1097--4571(198601)37:1-48::AID-ASI10>3.0.CO;2--3 · doi:10.1002/(sici)1097-4571(198601)37:1<48::aid-asi10>3.0.co;2-3
  10. William K. Frankena. 1939. The naturalistic fallacy. Mind 48, 192, Article 4 (1939), 13 pages. https://doi.org/10.1093/mind/XLVIII.192.464 · doi:10.1093/mind/xlviii.192.464
  11. Cecilie Gaziano and Kristin McGrath. 1986. Measuring the concept of credibility. Journalism quarterly 63, 3 (1986), 451--462. · DOI · Paper page
  12. Manish Gupta, Peixiang Zhao, and Jiawei Han. 2012. Evaluating event credibility on twitter. In Proceedings of the 2012 SIAM International Conference on Data Mining. SIAM, SIAM, 3600 Market Street, 6th Floor | Philadelphia, PA 19104--2688 USA, 153--164. · DOI · Paper page
  13. Miriam J. Metzger, Andrew J. Flanagin, Keren Eyal, Daisy R. Lemus, and Robert M. Mccann. 2003. Credibility for the 21st Century: Integrating Perspectives on Source, Message, and Media Credibility in the Contemporary Media Environment. Annals of the International Communication Association 27, 1 (2003), 293--335. https: / / doi.org /10.1080 /23808985.2003.11679029 arXiv:https://doi.org/10.1080/23808985.2003.11679029 · DOI · Paper page
  14. Hans K Meyer, Doreen Marchionni, and Esther Thorson. 2010. The journalist behind the news: credibility of straight, collaborative, opinionated, and blogged --news--. American Behavioral Scientist 54, 2 (2010), 100--119.
  15. Tanushree Mitra and Eric Gilbert. 2015. CREDBANK: A Large-Scale Social Media Corpus With Associated Credibility Annotations. In Ninth International AAAI Conference on Web and Social Media. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., Article 10582, 10 pages.
  16. Meredith Ringel Morris, Scott Counts, Asta Roseway, Aaron Hoff, and Julia Schwarz. 2012. Tweeting is believing: understanding microblog credibility perceptions. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 441--450. · DOI · Paper page
  17. Ryosuke Nagura, Yohei Seki, Noriko Kando, and Masaki Aono. 2006. A Method of Rating the Credibility of News Documents on the Web. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '06). ACM, New York, NY, USA, Article 1148316, 2 pages. https://doi.org/10.1145/1148170.1148316 · doi:10.1145/1148170.1148316
  18. Daniel J Okeefe. 2002. Persuasion: Theory and research. Vol. 2. Sage, Los Angeles, CA.
  19. Gordon Pennycook and David G Rand. 2017. Assessing the effect of disputed warnings and source salience on perceptions of fake news accuracy. Technical Report. SSRN. http://dx.doi.org/10.2139/ssrn.3035384 · DOI
  20. Peter Pirolli, Evelin Wollny, and Bongwon Suh. 2009. So you know youre getting the best possible information: a tool that increases Wikipedia credibility. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 1505--1508.
  21. Martin Potthast, Sebastian Köpsel, Benno Stein, and Matthias Hagen. 2016. Clickbait detection. In European Conference on Information Retrieval. Springer, Springer International Publishing, Gewerbestrasse 11, 6330 Cham, Switzerland, 810--817. · DOI
  22. Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. ACL, 209 N. Eighth Street, Stroudsburg PA 18360, USA, 2921--2927.
  23. Marta Reacasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic Models for Analyzing and Detecting Biased Language. In 51st Annual Meeting of the Association for Computational Linguistics. ACL, ACL, 209 N. Eighth Street, Stroudsburg PA 18360, USA, 1650--1659.
  24. Paul Resnick, Samuel Carton, Souneil Park, Yuncheng Shen, and Nicole Zeffer. 2014. Rumorlens: A system for analyzing the impact of rumors and corrections in social media. In Proc. Computational Journalism Conference. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 5.
  25. Mike Schmierbach and Anne Oeldorf-Hirsch. 2012. A little bird told me, so I didnt believe it: Twitter, credibility, and issue perceptions. Communication Quarterly 60, 3 (2012), 317--337. · DOI · Paper page
  26. Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake News Detection on Social Media: A Data Mining Perspective. SIGKDD Explor. Newsl. 19, 1, Article 3137600 (Sept. 2017), 15 pages. https://doi.org/10.1145/ 3137597.3137600 · DOI · Paper page
  27. Kate Starbird. 2017. Examining the Alternative Media Ecosystem Through the Production of Alternative Narratives of Mass Shooting Events on Twitter. In ICWSM. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 230--239.
  28. S Shyam Sundar. 1998. Effect of source attribution on perception of online news stories. Journalism & Mass Communication Quarterly 75, 1 (1998), 55--68. · DOI · Paper page
  29. Lauren Vogel. 2017. Viral misinformation threatens public health. Canadian Medical Association Journal 189, 50, Article E1567 (Dec 2017), 1 pages. https: //www.ncbi.nlm.nih.gov/pmc/articles/PMC5738254/ · DOI
  30. You Wu, Pankaj K Agarwal, Chengkai Li, Jun Yang, and Cong Yu. 2017. Computational Fact Checking through Query Perturbations. ACM Transactions on Database Systems (TODS) 42, 1 (2017), 4.
  31. Kenneth C.C. Yang. 2007. Factors influencing Internet users-- perceived credibility of news-related blogs in Taiwan. Telematics and Informatics 24, 2 (2007), 69--85. https://doi.org/10.1016/j.tele.2006.04.001 · DOI
  32. Wenlin Yao, Zeyu Dai, Ruihong Huang, and James Caverlee. 2017. Online Deception Detection Refueled by Real World Data Collection. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017. INCOMA Ltd., Varna, Bulgaria, 793--802. https://doi.org/10.26615/978- 954--452-049--6102 · doi:10.26615/978-

Cited by

See all 120

Metadata

Authors
Zhang, A. X., Robbins, M., Bice, E., Hawke, S., Karger, D., Mina, A. X., Ranganathan, A., Metz, S. E., Appling, S., Sehat, C. M., Gilmore, N., Adams, N. B., Vincent, E., & Lee, J.
Venue
Companion of the The Web Conference 2018 on The Web Conference 2018 - WWW '18
Publisher
ACM Press
Published
2018
Pages
603-612
OpenAlex
W2798818974
Cited by
120
References
10
FWCI
45.39
Type
conference paper
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Structured Response to Misinformation: Defining and Annotating Credibility Indicators in News Articles. https://app.cochat.ai/papers/10.1145/3184558.3188731
The paper
Zhang, A. X., Robbins, M., Bice, E., Hawke, S., Karger, D., Mina, A. X., Ranganathan, A., Metz, S. E., Appling, S., Sehat, C. M., Gilmore, N., Adams, N. B., Vincent, E., & Lee, J. (2018). A Structured Response to Misinformation: Defining and Annotating Credibility Indicators in News Articles. Companion of the The Web Conference 2018 on The Web Conference 2018 - WWW '18. https://doi.org/10.1145/3184558.3188731

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.