Evidence-based Factual Error Correction
Open accessAbstract
James Thorne, Andreas Vlachos. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The approach is inspired by works on fact-checking where simple research-and-revise workflows are effective at attributing or correcting unattributed claims made by humans.
“The approach is inspired by works on fact-checking 2 where simple research-and-revise workflows are effective at attributing or correcting unattributed claims made by humans (Thorne et al., 2018;Schuster et al., 2021;Thorne and Vlachos, 2021).”
From RARR: Researching and revising what language models say, using language models (2023) · cites this paper as Thorne and Vlachos, 2021
The Retrieved Evidence section reports that the system retrieves evidence and uses it to correct claims, with satisfactory corrections for some evaluated instances. The Conclusions and Future Work section says the evaluation used mutations introduced by human annotators and may not represent errors in real-world documents. The paper does not demonstrate attribution of unattributed claims, so that part and the broader implication about human-made claims are not supported; page numbers are unavailable in our copy. (LLM, conf=0.86)
Experimental — accuracy not yet measuredEFEC fine-tunes a T5-based model to revise text conditioned on multiple evidence snippets using both semi-supervised and fully-supervised approaches.
“EFEC fine-tunes a T5-based model to revise text conditioned on multiple evidence snippets using both semi-supervised and fully-supervised approaches.”
From RARR: Researching and revising what language models say, using language models (2023) · cites this paper as Thorne and Vlachos, 2021
Section 5.3 says the paper fine-tunes a T5 model to generate corrections from masked claims and evidence, and Section 4.1 specifies multiple retrieved evidence sentences. The paper calls its main training approach “distant supervision,” not semi-supervision; it also fine-tunes a separate fully supervised T5 model as a performance ceiling (Implementation, “Fully Supervised Ceiling”). (LLM, conf=0.87)
Experimental — accuracy not yet measuredEven if a revision preserves intent, it may still make superfluous modifications, such as reordering words, changing textual style, or including unnecessary additional information.
“However, even if a revision preserves intent, it may still make superfluous modifications, such as reordering words, changing textual style, or including unnecessary additional information (Thorne and Vlachos, 2021).”
From RARR: Researching and revising what language models say, using language models (2023) · cites this paper as Thorne and Vlachos, 2021
Section R2—Supported by Evidence says corrections should target errors in the original claim rather than introduce unrelated information; Section 7 Evaluation also checks whether unrelated information was added. These passages support the concern about unnecessary additions, but the copy does not discuss intent-preserving revisions that needlessly reorder words or change textual style. No page numbers are available in the indexed passages. (LLM, conf=0.83)
Experimental — accuracy not yet measuredEFEC's fully-supervised approach performed best in their experiments.
“We compare against their fully-supervised approach, which performed best in their experiments.”
From RARR: Researching and revising what language models say, using language models (2023) · cites this paper as Thorne and Vlachos, 2021
Evidence-based Factual Error Correction (EFEC) implements a full research-and-revise workflow trained on Wikipedia passages.
“Evidence-based Factual Error Correction (EFEC; Thorne and Vlachos, 2021) also implements a full research-and-revise workflow trained on Wikipedia passages (Thorne et al., 2018).”
From RARR: Researching and revising what language models say, using language models (2023) · cites this paper as Thorne and Vlachos, 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575 · see citing works
Object Hallucination in Image Captioning
Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T., & Saenko, K. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1437 · Cited by 420 · see citing works
Flagged first, then by citations
Research works printed but not linked: 30
- Pepa Atanasova, Jakob Grue Simonsen, Christina Li- oma, and Isabelle Augenstein. 2020. Generating fact checking explanations. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 7352-7364. Association for Computational Linguistics. · DOI
- Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Chris- tian Hansen, and Jakob Grue Simonsen. 2019. MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 4685-4697, Hong Kong, China. Association for Computational Lin- guistics. · DOI
- Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020. Factual Error Correction for Abstrac- tive Summarization Models. In Empirical Methods in Natural Language Processing, pages 6251-6258. · DOI · Paper page
- Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In International Conference on Learning Represen- tations.
- Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 1657-1668, Vancouver, Canada. Association for Computational Linguistics. · DOI
- Sarah Cohen, Chengkai Li, Jun Yang, and Cong Yu. 2011. Computational Journalism: a call to arms to database researchers. Proceedings of the 5th Biennial Conference on Innovative Data Systems Research (CIDR 2011) Asilomar, California, USA., (January):148-151.
- Na-Rae Han, Joel Tetreault, Soo-Hwa Lee, and Jin- Young Ha. 2010. Using an error-annotated learner corpus to develop an ESL/EFL error correction sys- tem. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10), Valletta, Malta. European Language Re- sources Association (ELRA).
- Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9(8):1735-1780. · DOI · Paper page
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference on Learn- ing Representations.
- Georgi Karadzhov, Preslav Nakov, Lluís Màrquez, Alberto Barrón-Cedeño, and Ivan Koychev. 2017. Fully automated fact checking using external sources. In Proceedings of the International Confer- ence Recent Advances in Natural Language Process- ing, RANLP 2017, pages 344-353. · DOI
- Kevin Knight and Ishwar Chander. 1994. Automated postediting of documents. In Proceedings of the National Conference on Artificial Intelligence, vol- ume 1, pages 779-784.
- Neema Kotonya and Francesca Toni. 2020. Explain- able Automated Fact-Checking for Public Health Claims. In The 2020 Conference on Empirical Meth- ods in Natural Language Processing.
- Nayeon Lee, Belinda Li, Sinong Wang, Wen-tau Yih, Hao Ma, and Madian Khabsa. 2020. Language models as fact checkers? In Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER), volume 2, pages 36-41. Association for Computational Linguistics. · DOI
- Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics. · DOI
- Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christo- pher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natu- ral Language Learning: Shared Task, July, pages 1- 14. Association for Computational Linguistics. · DOI
- Feng Nie, Jin-Ge Yao, Jinpeng Wang, Rong Pan, and Chin-Yew Lin. 2019. A simple recipe towards re- ducing hallucination in neural surface realisation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2673-2679, Florence, Italy. Association for Compu- tational Linguistics. · DOI · Paper page
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic eval- uation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, July, pages 311-318. Associa- tion for Computational Linguistics. · DOI · Paper page
- Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP- IJCNLP), pages 2463-2473, Hong Kong, China. As- sociation for Computational Linguistics. · DOI · Paper page
- Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text- to-Text Transformer. Journal of Machine Learning Research, 21:1-67.
- Marco Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "why should I trust you?": Explaining the pre- dictions of any classifier. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demon- strations, volume 39, pages 97-101. Association for Computational Linguistics. · DOI · Paper page
- Abigail See, Peter J. Liu, and Christopher D. Man- ning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), vol- ume 1, pages 1073-1083. Association for Computa- tional Linguistics. · DOI · Paper page
- Darsh J Shah, Tal Schuster, and Regina Barzilay. 2020. Automatic Fact-guided Sentence Modification. In Proceedings of the AAAI Conference on Artificial In- telligence.
- Dominik Stammbach and Elliott Ash. 2020. e-FEVER: Explanations and Summaries for Automated Fact Checking. In Proceedings of the 2020 Truth and Trust Online Conference (TTO 2020), page 32. Hacks Hackers.
- Wilson L. Taylor. 1953. "Cloze Procedure": A New Tool for Measuring Readability. Journalism Quar- terly, 30(4):415-433. · DOI · Paper page
- James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809-819, New Orleans, Louisiana. Association for Computational Linguistics. · DOI · Paper page
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Lilon Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA. · DOI · Paper page
- William Yang Wang. 2017. "liar, liar pants on fire": A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422-426. Association for Computational Linguistics. · DOI
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-Art Natural Language Process- ing. In Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Asso- ciation for Computational Linguistics. · DOI · Paper page
- Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401-415. · DOI
- Chunting Zhou, Jiatao Gu, Mona Diab, Paco Guzman, Luke Zettlemoyer, and Marjan Ghazvininejad. 2020. Detecting Hallucinated Content in Conditional Neu- ral Sequence Generation. pages 1-21.
Cited by
Sorted by citations
RARR: Researching and revising what language models say, using language models
Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A. T., Fan, Y., Zhao, V., Lao, N., Lee, H., Juan, D.-C., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.910 · Cited by 141 · see citing works
Kamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00713 · Cited by 82 · see citing works
DialFact: A Benchmark for Fact-Checking in Dialogue
Gupta, P., Wu, C.-S., Liu, W., & Xiong, C. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.263 · Cited by 48 · see citing works
SciFact-Open: Towards open-domain scientific claim verification
Wadden, D., Lo, K., Kuehl, B., Cohan, A., Beltagy, I., Wang, L. L., & Hajishirzi, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.347 · Cited by 35 · see citing works
Eldifrawi, I., Wang, S., & Trabelsi, A. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.361 · Cited by 5 · see citing works
Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers
Wang, Y., Reddy, R. G., Mujahid, Z. M., Arora, A., Rubashevskii, A., Geng, J., Afzal, O. M., Pan, L., Borenstein, N., Pillai, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.09000 · Cited by 3 · see citing works
- Authors
- Thorne, J., & Vlachos, A.
- Affiliations
- University of Cambridge (GB)
- ORCID
- Vlachos, Andreas
- Venue
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Published
- 2021
- Pages
- 3298-3309
- OpenAlex
- W3169467202
- Cited by
- 39
- References
- 3
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 4 · 2023: 11 · 2022: 10
- FWCI
- 2.89
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- arXiv (Cornell University) (submittedVersion)
- Apollo (University of Cambridge) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Apollo (cc-by)
- Funders
- H2020 European Research Council (865958)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- factual error correction, factual errors, factual knowledge, factual reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
