Predicting Factuality of Reporting and Bias of News Media Sources
Open accessThe DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
We present a study on predicting the factuality of reporting and bias of news media.While previous work has focused on studying the veracity of claims or documents, here we are interested in characterizing entire news media.These are under-studied but arguably important research problems, both in their own right and as a prior for fact-checking systems.We experiment with a large list of news websites and with a rich set of features derived from (i) a sample of articles from the target news medium, (ii) its Wikipedia page, (iii) its Twitter account, (iv) the structure of its URL, and (v) information about the Web traffic it attracts.The experimental results show sizable performance gains over the baselines, and confirm the importance of each feature type.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Liberals and conservatives rely on different sets of moral foundations.
Graham, J., Haidt, J., & Nosek, B. A. (2009) · Journal of Personality and Social Psychology · doi:10.1037/a0015141 · Cited by 4,560
Lazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., et al. (2018) · Science · doi:10.1126/science.aao2998 · Cited by 4,160
FEVER: a Large-scale Dataset for Fact Extraction and VERification
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1074 · Cited by 1,343 · cited 3×
“, 2018), and the FEVER-2018 task on Fact Extraction and VERification (Thorne et al., 2018).”
BackgroundMethodBaeza-Yates, R. (2018) · Communications of the ACM · doi:10.1145/3209581 · Cited by 432
Stop clickbait: detecting and preventing clickbaits in online news media
Chakraborty, A., Paranjape, B., Kakarla, S., & Ganguly, N. (2016) · 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM) · doi:10.1109/asonam.2016.7752207 · Cited by 368 · cited 2×
“…: POS tags, linguistic features based on the use of specific words (function words, pronouns, etc.), and features for clickbait title classification from (Chakraborty et al., 2016); • Sentiment : sentiment scores using lexicons (Recasens et al., 2013; Mitchell et al., 2013) and full systems (Hutto…”
MethodEvaluation Measures for Ordinal Regression
Baccianella, S., Esuli, A., & Sebastiani, F. (2009) · 2009 Ninth International Conference on Intelligent Systems Design and Applications · doi:10.1109/isda.2009.230 · Cited by 222
Cited by
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
A Survey of Fake News: Fundamental Theories, Detection Methods, and Opportunities
Zhou, X., & Zafarani, R. (2021) · ACM Computing Surveys · doi:10.1145/3395046 · Cited by 1,184
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2023) · arXiv · doi:10.48550/arxiv.2305.03514 · Cited by 839
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904
“A related line of research is automated fact-checking (Thorne et al., 2018; Aly et al., 2021; Baly et al., 2018), where the focus is on evaluation of statements rather than generation.”
BackgroundRealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585
Fake news detection based on news content and social contexts: a transformer-based approach
Raza, S., & Ding, C. (2022) · International Journal of Data Science and Analytics · doi:10.1007/s41060-021-00302-z · Cited by 285
FANG: Leveraging Social Context for Fake News Detection Using Graph Representation
Nguyen, V.-H., Sugiyama, K., Nakov, P., & Kan, M.-Y. (2020) · Proceedings of the 29th ACM International Conference on Information & Knowledge Management · doi:10.1145/3340531.3412046 · Cited by 270
Self-citationFactuality challenges in the era of large language models and opportunities for fact-checking
Augenstein, I., Baldwin, T., Cha, M., Chakraborty, T., Ciampaglia, G. L., Corney, D., DiResta, R., Ferrara, E., Hale, S., Halevy, A., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00881-z · Cited by 187
“This requires a comprehensive strategy that includes identifying and verifying the credibility of the sources 108–111 , ensuring that the content is based on verifiable data with evidence-based explanatory artefacts, dedication to truthfulness and effectively communicating factual corrections to…”
Self-citationPaschen, J. (2019) · Journal of Product & Brand Management · doi:10.1108/jpbm-12-2018-2179 · Cited by 132
Feature-based detection of automated language models: tackling GPT-2, GPT-3 and Grover
Fröhling, L., & Zubiaga, A. (2021) · PeerJ Computer Science · doi:10.7717/peerj-cs.443 · Cited by 106
InfoSurgeon: Cross-Media Fine-grained Information Consistency Checking for Fake News Detection
Fung, Y., Thomas, C., Gangi Reddy, R., Polisetty, S., Ji, H., Chang, S.-F., McKeown, K., Bansal, M., & Sil, A. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.133 · Cited by 52
Dataset of Fake News Detection and Fact Verification: A Survey
Murayama, T. (2021) · arXiv · doi:10.48550/arxiv.2111.03299 · Cited by 40
“MediaReliability [32] includes 1,066 news media information such as a sample of articles, its Wikipedia page, its Twitter account, and so on, for a study on predicting the factuality and bias of news media.”
Background(2019) · doi:10.18653/v1/d19-1292 · Cited by 36
Srba, I., Razuvayevskaya, O., Leite, J. A., Moro, R., Schlicht, I. B., Tonelli, S., García, F. M., Lottmann, S. B., Teyssou, D., Porcellini, V., et al. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3770077 · Cited by 19
One-Class Learning for AI-Generated Essay Detection
Corizzo, R., & Leal-Arenas, S. (2023) · Applied Sciences · doi:10.3390/app13137901 · Cited by 16
Can We Spot the "Fake News" Before It Was Even Written?
Nakov, P. (2020) · arXiv · doi:10.48550/arxiv.2008.04374 · Cited by 15
Self-citationA deep fusion model for human $vs$. Machine-generated essay classification
Corizzo, R., & Leal-Arenas, S. (2023) · 2023 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191322 · Cited by 13
Guo, Y., Tong, J., Zhao, J., Pan, W., Du, X., Hu, J., Yang, Q., Wen, F., & Ren, Z. (2025) · Journal of Air Transport Management · doi:10.1016/j.jairtraman.2025.102858
Metadata
- Authors
- Baly, R., Karadzhov, G., Alexandrov, D., Glass, J., & Nakov, P.
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 3528-3539
- OpenAlex
- W2891946729
- Cited by
- 254
- References
- 6
- FWCI
- 97.70
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.