A Structured Response to Misinformation: Defining and Annotating Credibility Indicators in News Articles
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
The proliferation of misinformation in online news and its amplification by platforms are a growing concern, leading to numerous efforts to improve the detection of and response to misinformation. Given the variety of approaches, collective agreement on the indicators that signify credible content could allow for greater collaboration and data-sharing across initiatives. In this paper, we present an initial set of indicators for article credibility defined by a diverse coalition of experts. These indicators originate from both within an article's text as well as from external sources or article metadata. As a proof-of-concept, we present a dataset of 40 articles of varying credibility annotated with our indicators by 6 trained annotators using specialized platforms. We discuss future steps including expanding annotation, broadening the set of indicators, and considering their use by platforms and the public, towards the development of interoperable standards for content credibility.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why the impact factor of journals should not be used for evaluating research
Seglen, P. O. (1997) · BMJ · doi:10.1136/bmj.314.7079.497 · Cited by 2,447
McCright, A. M., & Dunlap, R. E. (2011) · The Sociological Quarterly · doi:10.1111/j.1533-8525.2011.01198.x · Cited by 2,340
Twitter under crisis: can we trust what we RT?
Mendoza, M., Poblete, B., & Castillo, C. (2010) · Proceedings of the First Workshop on Social Media Analytics · doi:10.1145/1964858.1964869 · Cited by 955
Ratkiewicz (2011) · doi:10.1145/1963192.1963301 · Cited by 541 · cited 1×
“Building on this work, researchers have built models to predict credibility of social media posts [6, 17, 22], as well as tools for investigating rumors or claims [23, 38, 40].”
BackgroundComputational Fact Checking from Knowledge Networks
Ciampaglia, G. L., Shiralkar, P., Rocha, L. M., Bollen, J., Menczer, F., & Flammini, A. (2015) · PLOS ONE · doi:10.1371/journal.pone.0128193 · Cited by 486
Credibility: A multidisciplinary framework
Rieh, S. Y., & Danielson, D. R. (2007) · Annual Review of Information Science and Technology · doi:10.1002/aris.2007.1440410114 · Cited by 336
Lateral reading: Reading less and learning more when evaluating digital information
Wineburg, S., & McGrew, S. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3048994 · Cited by 198
Evaluating the Credibility of Online Information: A Test of Source and Advertising Influence
Greer, J. D. (2003) · Mass Communication and Society · doi:10.1207/s15327825mcs0601_3 · Cited by 197
Credibility Assessment of Textual Claims on the Web
Popat, K., Mukherjee, S., Strötgen, J., & Weikum, G. (2016) · Proceedings of the 25th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2983323.2983661 · Cited by 177
Using TwitterTrails.com to Investigate Rumor Propagation
Metaxas, P. T., Finn, S., & Mustafaraj, E. (2015) · Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing · doi:10.1145/2685553.2702691 · Cited by 54 · cited 1×
“Building on this work, researchers have built models to predict credibility of social media posts [6, 17, 22], as well as tools for investigating rumors or claims [23, 38, 40].”
Background
Research works printed but not linked: 32
- Ahmer Arif, John J Robinson, Stephanie A Stanek, Elodie S Fichet, Paul Townsend, Zena Worku, and Kate Starbird. 2017. A Closer Look at the Self-Correcting Crowd: Examining Corrections in Online Rumors. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 155--168. · DOI · Paper page
- Ahmer Arif, Kelley Shanahan, Fang-Ju Chou, Yoanna Dosouto, Kate Starbird, and Emma S Spiro. 2016. How information snowballs: Exploring the role of exposure in online rumor propagation. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 466--477.
- Leticia Bode and Emily K Vraga. 2015. In related news, that was wrong: The correction of misinformation through related stories functionality in social media. Journal of Communication 65, 4 (2015), 619--638. · DOI · Paper page
- David B. Buller and Judee K. Burgoon. 1996. Interpersonal Deception Theory. Communication Theory 6, 3 (1996), 203--242. https://doi.org/10.1111/j.1468- 2885.1996.tb00127.x · doi:10.1111/j.1468-
- Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. 2011. Information credibility on twitter. In Proceedings of the 20th international conference on World wide web. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 675--684. · DOI · Paper page
- Michael A. Caulfield. 2017. Go Upstream to the Find the Source. (Jan 2017). https://webliteracy.pressbooks.com/chapter/go-upstream-to-find-the-source/
- Yimin Chen, Niall J Conroy, and Victoria L Rubin. 2015. Misleading online content: Recognizing clickbait as false news. In Proceedings of the 2015 ACM on Workshop on Multimodal Deception Detection. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 15--19.
- Andrew J Flanagin and Miriam J Metzger. 2000. Perceptions of Internet information credibility. Journalism & Mass Communication Quarterly 77, 3 (2000), 515--540. · DOI · Paper page
- William B. Frakes. 1986. Information and misinformation: An investigation of the notions of information, misinformation, informing, and misinforming. Journal of the American Society for Information Science 37, 1 (1986), 48--49. https: //doi.org/10.1002/(SICI)1097--4571(198601)37:1-48::AID-ASI10>3.0.CO;2--3 · doi:10.1002/(sici)1097-4571(198601)37:1<48::aid-asi10>3.0.co;2-3
- William K. Frankena. 1939. The naturalistic fallacy. Mind 48, 192, Article 4 (1939), 13 pages. https://doi.org/10.1093/mind/XLVIII.192.464 · doi:10.1093/mind/xlviii.192.464
- Cecilie Gaziano and Kristin McGrath. 1986. Measuring the concept of credibility. Journalism quarterly 63, 3 (1986), 451--462. · DOI · Paper page
- Manish Gupta, Peixiang Zhao, and Jiawei Han. 2012. Evaluating event credibility on twitter. In Proceedings of the 2012 SIAM International Conference on Data Mining. SIAM, SIAM, 3600 Market Street, 6th Floor | Philadelphia, PA 19104--2688 USA, 153--164. · DOI · Paper page
- Miriam J. Metzger, Andrew J. Flanagin, Keren Eyal, Daisy R. Lemus, and Robert M. Mccann. 2003. Credibility for the 21st Century: Integrating Perspectives on Source, Message, and Media Credibility in the Contemporary Media Environment. Annals of the International Communication Association 27, 1 (2003), 293--335. https: / / doi.org /10.1080 /23808985.2003.11679029 arXiv:https://doi.org/10.1080/23808985.2003.11679029 · DOI · Paper page
- Hans K Meyer, Doreen Marchionni, and Esther Thorson. 2010. The journalist behind the news: credibility of straight, collaborative, opinionated, and blogged --news--. American Behavioral Scientist 54, 2 (2010), 100--119.
- Tanushree Mitra and Eric Gilbert. 2015. CREDBANK: A Large-Scale Social Media Corpus With Associated Credibility Annotations. In Ninth International AAAI Conference on Web and Social Media. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., Article 10582, 10 pages.
- Meredith Ringel Morris, Scott Counts, Asta Roseway, Aaron Hoff, and Julia Schwarz. 2012. Tweeting is believing: understanding microblog credibility perceptions. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 441--450. · DOI · Paper page
- Ryosuke Nagura, Yohei Seki, Noriko Kando, and Masaki Aono. 2006. A Method of Rating the Credibility of News Documents on the Web. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '06). ACM, New York, NY, USA, Article 1148316, 2 pages. https://doi.org/10.1145/1148170.1148316 · doi:10.1145/1148170.1148316
- Daniel J Okeefe. 2002. Persuasion: Theory and research. Vol. 2. Sage, Los Angeles, CA.
- Gordon Pennycook and David G Rand. 2017. Assessing the effect of disputed warnings and source salience on perceptions of fake news accuracy. Technical Report. SSRN. http://dx.doi.org/10.2139/ssrn.3035384 · DOI
- Peter Pirolli, Evelin Wollny, and Bongwon Suh. 2009. So you know youre getting the best possible information: a tool that increases Wikipedia credibility. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 1505--1508.
- Martin Potthast, Sebastian Köpsel, Benno Stein, and Matthias Hagen. 2016. Clickbait detection. In European Conference on Information Retrieval. Springer, Springer International Publishing, Gewerbestrasse 11, 6330 Cham, Switzerland, 810--817. · DOI
- Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. ACL, 209 N. Eighth Street, Stroudsburg PA 18360, USA, 2921--2927.
- Marta Reacasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic Models for Analyzing and Detecting Biased Language. In 51st Annual Meeting of the Association for Computational Linguistics. ACL, ACL, 209 N. Eighth Street, Stroudsburg PA 18360, USA, 1650--1659.
- Paul Resnick, Samuel Carton, Souneil Park, Yuncheng Shen, and Nicole Zeffer. 2014. Rumorlens: A system for analyzing the impact of rumors and corrections in social media. In Proc. Computational Journalism Conference. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 5.
- Mike Schmierbach and Anne Oeldorf-Hirsch. 2012. A little bird told me, so I didnt believe it: Twitter, credibility, and issue perceptions. Communication Quarterly 60, 3 (2012), 317--337. · DOI · Paper page
- Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake News Detection on Social Media: A Data Mining Perspective. SIGKDD Explor. Newsl. 19, 1, Article 3137600 (Sept. 2017), 15 pages. https://doi.org/10.1145/ 3137597.3137600 · DOI · Paper page
- Kate Starbird. 2017. Examining the Alternative Media Ecosystem Through the Production of Alternative Narratives of Mass Shooting Events on Twitter. In ICWSM. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 230--239.
- S Shyam Sundar. 1998. Effect of source attribution on perception of online news stories. Journalism & Mass Communication Quarterly 75, 1 (1998), 55--68. · DOI · Paper page
- Lauren Vogel. 2017. Viral misinformation threatens public health. Canadian Medical Association Journal 189, 50, Article E1567 (Dec 2017), 1 pages. https: //www.ncbi.nlm.nih.gov/pmc/articles/PMC5738254/ · DOI
- You Wu, Pankaj K Agarwal, Chengkai Li, Jun Yang, and Cong Yu. 2017. Computational Fact Checking through Query Perturbations. ACM Transactions on Database Systems (TODS) 42, 1 (2017), 4.
- Kenneth C.C. Yang. 2007. Factors influencing Internet users-- perceived credibility of news-related blogs in Taiwan. Telematics and Informatics 24, 2 (2007), 69--85. https://doi.org/10.1016/j.tele.2006.04.001 · DOI
- Wenlin Yao, Zeyu Dai, Ruihong Huang, and James Caverlee. 2017. Online Deception Detection Refueled by Real World Data Collection. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017. INCOMA Ltd., Varna, Bulgaria, 793--802. https://doi.org/10.26615/978- 954--452-049--6102 · doi:10.26615/978-
Cited by
Lai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377
“[80], which explores varying types of credibility annotations specifically designed for news articles.”
BackgroundZhou, J., Zhang, Y., Luo, Q., Parker, A. G., & De Choudhury, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581318 · Cited by 288
Jahanbakhsh, F., Zhang, A. X., Berinsky, A. J., Pennycook, G., Rand, D. G., & Karger, D. R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449092 · Cited by 112
“…detecting false or misleading information using machine learning algorithms [12, 57, 67] and crowdsourcing [5, 9, 19, 33, 54], identifying bad actors and helping good actors differentiate themselves 1 [82], and providing fact-checked information related to circulated news claims [23, 35, 50, 81].”
BackgroundInfluentialSelf-citationMisinformation on Instagram: The Impact of Trusted Endorsements on Message Credibility
Mena, P., Barbe, D., & Chan-Olmsted, S. (2020) · Social Media + Society · doi:10.1177/2056305120935102 · Cited by 102
The Web of False Information: Rumors, Fake News, Hoaxes, Clickbait, and Various Other Shenanigans
Zannettou, S., Sirivianos, M., Blackburn, J., & Kourtellis, N. (2019) · Journal of Data and Information Quality · doi:10.1145/3309699 · Cited by 86
Metadata
- Authors
- Zhang, A. X., Robbins, M., Bice, E., Hawke, S., Karger, D., Mina, A. X., Ranganathan, A., Metz, S. E., Appling, S., Sehat, C. M., Gilmore, N., Adams, N. B., Vincent, E., & Lee, J.
- Venue
- Companion of the The Web Conference 2018 on The Web Conference 2018 - WWW '18
- Publisher
- ACM Press
- Published
- 2018
- Pages
- 603-612
- OpenAlex
- W2798818974
- Cited by
- 120
- References
- 10
- FWCI
- 45.39
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Structured Response to Misinformation: Defining and Annotating Credibility Indicators in News Articles. https://app.cochat.ai/papers/10.1145/3184558.3188731
- The paper
- Zhang, A. X., Robbins, M., Bice, E., Hawke, S., Karger, D., Mina, A. X., Ranganathan, A., Metz, S. E., Appling, S., Sehat, C. M., Gilmore, N., Adams, N. B., Vincent, E., & Lee, J. (2018). A Structured Response to Misinformation: Defining and Annotating Credibility Indicators in News Articles. Companion of the The Web Conference 2018 on The Web Conference 2018 - WWW '18. https://doi.org/10.1145/3184558.3188731
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.