CoChat
Sign Up
Cited by 18Misinformation and Its ImpactsMultimodal Machine Learning ApplicationsDigital Media Forensic Detection

Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking

Open access
Akhtar, Mubashara, Cocarascu, Oana, Simperl, Elena·2023·Findings of the Association for Computational Linguistics: EACL 2023
DOI 10.18653/v1/2023.findings-eacl.30
Open this paper in CoChat — free accountOpen PDF

Abstract

Evidence data for automated fact-checking (AFC) can be in multiple modalities such as text, tables, images, audio, or video.While there is increasing interest in using images for AFC, previous works mostly focus on detecting manipulated or fake images.We propose a novel task, chart-based fact-checking, and introduce ChartBERT as the first model for AFC against chart evidence.ChartBERT leverages textual, structural and visual information of charts to determine the veracity of textual claims.For evaluation, we create ChartFC, a new dataset of 15, 886 charts.We systematically evaluate 75 different vision-language (VL) baselines and show that ChartBERT outperforms VL models, achieving 63.8% accuracy.Our results suggest that the task is complex yet feasible, with many challenges ahead.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2023.findings-eacl.30

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 50

  1. Sara Abdali. 2022. Multi-modal misinformation de- tection: Approaches, challenges and opportunities. CoRR, abs/2203.13883. · DOI
  2. Mubashara Akhtar, Oana Cocarascu, and Elena Sim- perl. 2022. PubHealthTab: A public health table- based dataset for evidence-based fact checking. In Findings of the Association for Computational Lin- guistics: NAACL 2022, pages 1-16, Seattle, United States. Association for Computational Linguistics. · DOI
  3. Firoj Alam, Stefano Cresci, Tanmoy Chakraborty, Fab- rizio Silvestri, Dimiter Dimitrov, Giovanni Da San Martino, Shaden Shaar, Hamed Firooz, and Preslav Nakov. 2021. A survey on multimodal disinforma- tion detection. CoRR, abs/2103.12541. · arXiv
  4. Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. FEVEROUS: fact extraction and verification over unstructured and structured information. In Proceedings of the Neural Infor- mation Processing Systems Track on Datasets and Benchmarks 1, NeurIPS.
  5. Robert A. Amar, James Eagan, and John T. Stasko. 2005. Low-level components of analytic activity in information visualization. In IEEE Symposium on Information Visualization (InfoVis), pages 111-117. IEEE Computer Society. · DOI
  6. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hin- ton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. · arXiv
  7. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR.
  8. Tadas Baltrušaitis, Chaitanya Ahuja, and Louis- Philippe Morency. 2018. Multimodal machine learn- ing: A survey and taxonomy. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 41(2):423-443.
  9. Michael Behrisch, Michael Blumenschein, Nam Wook Kim, Lin Shao, Mennatallah El-Assady, Johannes Fuchs, Daniel Seebacher, Alexandra Diehl, Ulrik Brandes, Hanspeter Pfister, Tobias Schreck, Daniel Weiskopf, and Daniel A. Keim. 2018. Quality met- rics for information visualization. Comput. Graph. Forum, 37(3):625-662. · DOI
  10. Efrat Blaier, Itzik Malkiel, and Lior Wolf. 2021. Cap- tion enriched samples for improving hateful memes detection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9350-9358, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics. · DOI
  11. Juan Cao, Peng Qi, Qiang Sheng, Tianyun Yang, Junbo Guo, and Jintao Li. 2020. Exploring the role of visual content in fake news detection. CoRR, abs/2003.05096. · arXiv
  12. Moses Charikar, Kevin C. Chen, and Martin Farach- Colton. 2004. Finding frequent items in data streams. Theor. Comput. Sci., 312(1):3-15. · DOI
  13. Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, and Ajay Joshi. 2020. LEAF-QA: locate, encode & attend for fig- ure question answering. In IEEE Winter Conference on Applications of Computer Vision, WACV, pages 3501-3510. IEEE. · DOI
  14. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020b. UNITER: universal image- text representation learning. In Computer Vision - ECCV -16th European Conference, volume 12375 of Lecture Notes in Computer Science, pages 104- 120. Springer. · DOI · Paper page
  15. Thomas Diggelmann, Jordan L. Boyd-Graber, Jan- nis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. CLIMATE-FEVER: A dataset for verification of real-world climate claims. CoRR, abs/2012.00614. · arXiv
  16. Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. 2016. Multimodal compact bilinear pooling for vi- sual question answering and visual grounding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 457-468, Austin, Texas. Association for Computa- tional Linguistics. · DOI · Paper page
  17. Ashim Gupta and Vivek Srikumar. 2021. X-fact: A new benchmark dataset for multilingual fact check- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 2: Short Papers), pages 675-682, Online. Association for Computa- tional Linguistics. · DOI
  18. Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as semi-structured data. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 2309-2324, Online. Association for Computational Linguistics. · DOI
  19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780. · DOI · Paper page
  20. Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. 2017. Densely connected con- volutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pages 2261-2269. IEEE Computer Society. · DOI · Paper page
  21. Kushal Kafle, Brian L. Price, Scott Cohen, and Christo- pher Kanan. 2018. DVQA: understanding data vi- sualizations via question answering. In 2018 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR, pages 5648-5656. Computer Vision Foundation / IEEE Computer Society. · DOI
  22. Kushal Kafle, Robik Shrestha, Brian L. Price, Scott Cohen, and Christopher Kanan. 2020. Answering questions about data visualizations using efficient bi- modal fusion. In IEEE Winter Conference on Ap- plications of Computer Vision, WACV, pages 1487- 1496. IEEE. · DOI
  23. Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2018. Figureqa: An annotated figure dataset for visual reasoning. In 6th International Conference on Learning Representations, ICLR. OpenReview.net.
  24. Shankar Kantharaj, Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022. Chart-to-text: A large-scale benchmark for chart summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4005-4023, Dublin, Ireland. Association for Computational Linguistics. · DOI
  25. Anthony Kay. 2007. Tesseract: an open-source op- tical character recognition engine. Linux Journal, 2007(159):2.
  26. Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. 2020. The hateful memes chal- lenge: Detecting hate speech in multimodal memes. In Advances in Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS.
  27. Dae Hyun Kim, Enamul Hoque, and Maneesh Agrawala. 2020. Answering questions about charts and generating visual explanations. In CHI '20: CHI Conference on Human Factors in Computing Systems, pages 1-13. ACM. · DOI
  28. Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-language transformer without convolu- tion or region supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 5583-5594. PMLR.
  29. Neema Kotonya and Francesca Toni. 2020. Ex- plainable automated fact-checking for public health claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 7740-7754, Online. Associa- tion for Computational Linguistics. · DOI · Paper page
  30. Crystal Lee, Tanya Yang, Gabrielle D. Inchoco, Gra- ham M. Jones, and Arvind Satyanarayan. 2021. Vi- ral visualizations: How coronavirus skeptics use or- thodox data practices to promote unorthodox science online. In CHI '21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yoko- hama, Japan, May 8-13, 2021, pages 607:1-607:18. ACM. · DOI
  31. Leo Yu-Ho Lo, Ayush Gupta, Kento Shigyo, Aoyu Wu, Enrico Bertini, and Huamin Qu. 2022. Mis- informed by visualization: What do we learn from misinformative visualizations? Comput. Graph. Fo- rum, 41(3):515-525. · DOI
  32. Preslav Nakov, David P. A. Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barrón-Cedeño, Paolo Papotti, Shaden Shaar, and Giovanni Da San Martino. 2021a. Automated fact-checking for assist- ing human fact-checkers. In Proceedings of the Thir- tieth International Joint Conference on Artificial In- telligence, IJCAI, pages 4551-4558. ijcai.org. · DOI
  33. Preslav Nakov, Giovanni Da San Martino, Tamer Elsayed, Alberto Barrón-Cedeño, Rubén Míguez, Shaden Shaar, Firoj Alam, Fatima Haouari, Maram Hasanain, Nikolay Babulkov, Alex Nikolov, Gau- tam Kishore Shahi, Julia Maria Struß, and Thomas Mandl. 2021b. The CLEF-2021 CheckThat! lab on detecting check-worthy claims, previously fact- checked claims, and fake news. In Advances in In- formation Retrieval -43rd European Conference on IR Research, ECIR, volume 12657 of Lecture Notes in Computer Science, pages 639-649. Springer. · DOI
  34. Anshul Vikram Pandey, Anjali Manivannan, Oded Nov, Margaret Satterthwaite, and Enrico Bertini. 2014. The persuasive power of data visualization. IEEE Trans. Vis. Comput. Graph., 20(12):2211-2220. · DOI · Paper page
  35. Jingnong Qu, Liunian Harold Li, Jieyu Zhao, Sunipa Dev, and Kai-Wei Chang. 2022. Disinfomeme: A multimodal dataset for detecting meme inten- tionally spreading out disinformation. CoRR, abs/2205.12617. · DOI
  36. Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with con- trastive evidence. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 624-643, Online. Asso- ciation for Computational Linguistics. · DOI · Paper page
  37. Shivam Sharma, Firoj Alam, Md. Shad Akhtar, Dim- itar Dimitrov, Giovanni Da San Martino, Hamed Firooz, Alon Y. Halevy, Fabrizio Silvestri, Preslav Nakov, and Tanmoy Chakraborty. 2022. Detecting and understanding harmful memes: A survey. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 5597-5606. ijcai.org. · DOI
  38. Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 5100-5111, Hong Kong, China. Association for Computational Linguistics. · DOI · Paper page
  39. Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely. 2021. Representing numbers in NLP: a sur- vey and a vision. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 644-656, Online. Asso- ciation for Computational Linguistics. · DOI
  40. James Thorne, Max Glockner, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. 2021. Evidence- based verification for real world information needs. CoRR, abs/2104.00640. · arXiv
  41. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics. · DOI · Paper page
  42. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference on Neural Information Processing Systems, pages 5998-6008.
  43. Pavlos Vougiouklis, Leslie Carr, and Elena Simperl. 2020. Pie chart or pizza: Identifying chart types and their virality on twitter. In Proceedings of the Four- teenth International AAAI Conference on Web and Social Media, ICWSM 2020, Held Virtually, Origi- nal Venue: Atlanta, Georgia, USA, June 8-11, 2020, pages 694-704. AAAI Press.
  44. Nancy X. R. Wang, Diwakar Mahajan, Marina Danilevsky, and Sara Rosenthal. 2021a. SemEval- 2021 task 9: Fact verification and evidence finding for tabular data in scientific documents (SEM-TAB- FACTS). In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 317-326, Online. Association for Computa- tional Linguistics. · DOI
  45. Nancy Xin Ru Wang, Diwakar Mahajan, Marina Danilevsky, and Sara Rosenthal. 2021b. Semeval- 2021 task 9: Fact verification and evidence find- ing for tabular data in scientific documents (SEM- TAB-FACTS). In Proceedings of the 15th In- ternational Workshop on Semantic Evaluation, Se- mEval@ACL/IJCNLP, pages 317-326. Association for Computational Linguistics. · DOI
  46. William Yang Wang. 2017. "liar, liar pants on fire": A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 422-426, Vancouver, Canada. Association for Computational Linguistics. · DOI
  47. Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florêncio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. 2021. TAP: text-aware pre- training for text-vqa and text-caption. In IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR, pages 8751-8761. Computer Vision Foundation / IEEE.
  48. Barry Menglong Yao, Aditya Shah, Lichao Sun, Jin- Hee Cho, and Lifu Huang. 2022. End-to-end multimodal fact-checking and explanation genera- tion: A challenging dataset and models. CoRR, abs/2205.12487. · DOI
  49. Yixuan Zhang, Yifan Sun, Lace M. K. Padilla, Sumit Barua, Enrico Bertini, and Andrea G. Parker. 2021. Mapping the landscape of COVID-19 crisis visual- izations. In CHI '21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yoko- hama, Japan, May 8-13, 2021, pages 608:1-608:23. ACM. · DOI
  50. Dimitrina Zlatkova, Preslav Nakov, and Ivan Koychev. 2019. Fact-checking meets fauxtography: Verify- ing claims about images. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2099-2108, Hong Kong, China. Association for Computational Linguistics. · DOI

Cited by

Sorted by citations

Authors
Akhtar, M., Cocarascu, O., & Simperl, E.
Affiliations
King's College London (GB)
ORCID
Venue
Findings of the Association for Computational Linguistics: EACL 2023
Publisher
Association for Computational Linguistics
Published
2023
Pages
399-414
DOI
10.18653/v1/2023.findings-eacl.30
OpenAlex
W4386576698
Cited by
18
References
7
Citations by year
2026: 6 · 2025: 3 · 2024: 6 · 2023: 3
FWCI
14.03
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Findings of the Association for Computational Linguistics: EACL (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Research Portal (King's College London) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
Keywords
automated fact-checking, vision-language models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review