CoChat
Sign Up
Cited by 70Misinformation and Its ImpactsComplex Network Analysis TechniquesAcademic Publishing and Open Access

True or False: Studying the Work Practices of Professional Fact-Checkers

Open access
Micallef, Nicholas, Armacost, Vivienne, Memon, Nasir, Patil, Sameer·2022·Proceedings of the ACM on Human-Computer Interaction
DOI 10.1145/3512974
Open this paper in CoChat — free accountOpen PDF

Abstract

Misinformation has developed into a critical societal threat that can lead to disastrous societal consequences. Although fact-checking plays a key role in combating misinformation, relatively little research has empirically investigated work practices of professional fact-checkers. To address this gap, we conducted semi-structured interviews with 21 fact-checkers from 19 countries. The participants reported being inundated with information that needs filtering and prioritizing prior to fact-checking. The interviews surfaced a pipeline of practices fragmented across disparate tools that lack integration. Importantly, fact-checkers lack effective mechanisms for disseminating the outcomes of their efforts which prevents their work from fully achieving its potential impact. We found that the largely manual and labor intensive nature of current fact-checking practices is a barrier to scale. We apply these findings to propose a number of suggestions that can improve the effectiveness, efficiency, scale, and reach of fact-checking work and its outcomes.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3512974

Cites

110 of 117 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 23

  1. Bill Adair, Mark Stencel, Cathy Clabby, and Chengkai Li. 2019. The human touch in automated fact-checking. In Proceedings of the 2019 Computation + Journalism Symposium (Miami, FL, USA) (C+J 2019). 5 pages. https: //par.nsf.gov/servlets/purl/10111586
  2. Andrew Beers, Melinda McClure Haughey, Ahmer Arif, and Kate Starbird. 2020. Examining the digital toolsets of journalists reporting on disinformation. In Proceedings of the 2020 Computation + Journalism Symposium (Boston, MA, USA) (C+J 2020). 5 pages. https://cj2020.northeastern.edu/fles/2020/02/CJ_2020_paper_50.pdf
  3. Cristian Felix, Anshul Vikram Pandey, Enrico Bertini, Charles Ornstein, and Scott Klein. 2015. Revex: Visual Investigative Journalism with A Million Healthcare Reviews. In Proceedings of the 2015 Computation + Journalism Symposium (New York, NY, USA) (C+J 2015). 6 pages. http://cj2015.brown.columbia.edu/papers/revex.pdf
  4. Lucas Graves and Magda Konieczna. 2015. Sharing the news: Journalistic collaboration as feld repair. International Journal of Communication 9 (2015), 1966-1984. https://ijoc.org/index.php/ijoc/article/view/3381/1412
  5. Binxuan Huang and Kathleen M. Carley. 2020. Disinformation and Misinformation on Twitter during the Novel Coronavirus Outbreak. arXiv:2006.04278 [cs.SI] · arXiv
  6. Kathleen Hall Jamieson. 2013. Electing the President, 2012: The Insiders' View. University of Pennsylvania Press.
  7. Kathy Lee, Diana Palsetia, Ramanathan Narayanan, Md. Mostofa Ali Patwary, Ankit Agrawal, and Alok Choudhary. 2011. Twitter Trending Topic Classifcation. In 2011 IEEE 11th International Conference on Data Mining Workshops. 251-258. https://doi.org/10.1109/ICDMW.2011.171 · DOI
  8. Piroska Lendvai, Isabelle Augenstein, Kalina Bontcheva, and Thierry Declerck. 2016. Monolingual social media datasets for detecting contradiction and entailment. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC '16). 4602-4605. https://aclanthology.org/L16-1729
  9. Piroska Lendvai and Uwe D. Reichel. 2016. Contradiction Detection for Rumorous Claims. arXiv:1611.02588 [cs.CL] · arXiv
  10. Måns Magnusson, Jens Finnäs, and Leonard Wallentin. 2016. Finding the news lead in the data haystack: Automated local data journalism using crime data. In Proceedings of the 2016 Computation + Journalism Symposium (Stanford, CA, USA) (C+J 2016). 4 pages. https://journalism.stanford.edu/cj2016/fles/Finding%20the%20news%20lead%20in% 20the%20data%20haystack.pdf
  11. Hana Matatov, Adina Bechhofer, Lora Aroyo, Ofra Amir, and Mor Naaman. 2019. DejaVu: A System for Journalists to Collaboratively Address Visual Misinformation. In Proceedings of the 2019 Computation + Journalism Symposium (Miami, FL, USA) (C+J 2019). 5 pages.
  12. Armineh Nourbakhsh, Quanzhi Li, Xiaomo Liu, and Sameena Shah. 2017. "Breaking" Disasters: Predicting and Characterizing the Global News Value of Natural and Man-made Disasters. arXiv:1709.02510 [cs.SI] · arXiv
  13. Titus Plattner, Didier Orel, and Olivier Steiner. 2016. Flexible data scraping, multi-language indexing, entity extraction and taxonomies: Tadam, a Swiss tool to deal with huge amounts of unstructured data. In Proceedings of the 2016 Compu- tation + Journalism Symposium (Stanford, CA, USA) (C+J 2016). 5 pages. https://journalism.stanford.edu/cj2016/fles/ Flexible%20data%20scraping%20multi-language%20indexing%20entity%20extraction%20and%20taxonomies.pdf
  14. Paul Resnick, Samuel Carton, Souneil Park, Yuncheng Shen, and Nicole Zefer. 2014. RumorLens: A system for Analyzing the Impact of Rumors and Corrections in Social Media. In Proceedings of the 2014 Computation + Journalism Symposium (New York, NY, USA) (C+J 2014). 5 pages.
  15. Ronald E. Robertson and Christo Wilson. 2020. WebSearcher: Tools for Auditing Web Search. In Proceedings of the 2020 Computation + Journalism Symposium (Boston, MA, USA) (C+J 2020). 4 pages. https://cj2020.northeastern.edu/ fles/2020/02/CJ_2020_paper_38.pdf
  16. Karishma Sharma, Sungyong Seo, Chuizheng Meng, Sirisha Rambhatla, and Yan Liu. 2020. COVID-19 on Social Media: Analyzing Misinformation in Twitter Conversations. arXiv:2003.12309 [cs.SI] · arXiv
  17. Matt Shearer, Basile Simon, and Clement Geiger. 2014. Datastringer: Easy dataset monitoring for journalists. In Proceedings of the 2014 Computation + Journalism Symposium (New York, NY, USA) (C+J 2014). 5 pages.
  18. Fred S. Siebert, Theodore Peterson, and Wilbur Schramm. 1956. Four Theories of the Press: The Authoritarian, Libertarian, Social Responsibility, and Soviet Communist Concepts of What the Press Should Be and Do. University of Illinois Press.
  19. Anselm Strauss and Juliet M. Corbin. 1997. Grounded Theory in Practice. SAGE Publications, Inc.
  20. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: A large-scale dataset for Fact Extraction and VERifcation. arXiv:1803.05355 [cs.CL] · arXiv
  21. Annamarie Van Wyk. 2017. Fact-checking in the Global South: Facts about non-proft journalism funding models -A case study. Master's thesis. Stellenbosch University. http://hdl.handle.net/10019.1/101020
  22. Arkaitz Zubiaga, Elena Kochkina, Maria Liakata, Rob Procter, and Michal Lukasik. 2016. Stance Classifcation in Rumours as a Sequential Task Exploiting the Tree Structure of Social Media Conversations. arXiv:1609.09028 [cs.CL] · arXiv
  23. Arkaitz Zubiaga, Maria Liakata, and Rob Procter. 2016. Learning Reporting Dynamics during Breaking News for Rumour Detection in Social Media. arXiv:1610.07363 [cs.CL] · arXiv

Cited by

Sorted by citations

Related

Authors
Micallef, N., Armacost, V., Memon, N., & Patil, S.
Affiliations
  • New York University Abu Dhabi (AE)
  • Indiana University Bloomington (US)
  • New York University (US)
  • University of Utah (US)
ORCID
Venue
Proceedings of the ACM on Human-Computer Interaction
Publisher
Association for Computing Machinery (ACM)
Published
2022-03-30
Volume
6
Issue
CSCW1
Pages
1-44
ISSN
2573-0142 (ISSN-L)
DOI
10.1145/3512974
OpenAlex
W4225879562
Cited by
70
References
113
Citations by year
2026: 17 · 2025: 20 · 2024: 23 · 2023: 8 · 2022: 2
FWCI
19.07
Type
journal article
Language
en
Open access
Licence
Locations
Proceedings of the ACM on Human-Computer Interaction (publishedVersion, cc-by-nc-sa)
Funders
Indiana University Bloomington
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Complex Network Analysis Techniques
  • Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
Keywords
fact-checking, misinformation, work practices, information filtering, priority setting, scalability, manual labor, human-in-the-loop

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review