Rome was built in 1776: A Case Study on Factual Correctness in Knowledge-Grounded Response Generation
Open accessAbstract
Recently neural response generation models have leveraged large pre-trained transformer models and knowledge snippets to generate relevant and informative responses. However, this does not guarantee that generated responses are factually correct. In this paper, we examine factual correctness in knowledge-grounded neural response generation models. We present a human annotation setup to identify three different response types: responses that are factually consistent with respect to the input knowledge, responses that contain hallucinated knowledge, and non-verifiable chitchat style responses. We use this setup to annotate responses generated using different stateof-the-art models, knowledge snippets, and decoding strategies. In addition, to facilitate the development of a factual consistency detector, we automatically create a new corpus called Conv-FEVER that is adapted from the Wizard of Wikipedia dataset and includes factually consistent and inconsistent responses. We demonstrate the benefit of our Conv-FEVER dataset by showing that the models trained on this data perform reasonably well to detect factually inconsistent responses with respect to the provided knowledge through evaluation on our human annotated data. We will release the Conv-FEVER dataset and the human annotated responses.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Human evaluation is still one of the most commonly used approaches for evaluating hallucinations in NLG due to the challenging and imperfect nature of current automatic evaluation.
“Due to the challenging and imperfect nature of the current automatic evaluation of hallucinations in NLG, human evaluation [226, 233] is still one of the most commonly used approaches.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [226]
Section 3.1 (p. 3) describes human annotation of hallucinations, and Section 5 (p. 9) reports that the automatic detector’s performance varies across conditions. The paper does not establish that human evaluation is one of the most commonly used approaches in NLG, or explicitly attribute its use to the shortcomings of automatic evaluation. (LLM, conf=0.82)
Experimental — accuracy not yet measuredChatbots are usually evaluated by humans on factual consistency or factual correctness.
“Therefore, chatbots are usually evaluated by humans on factual consistency or factual correctness [226, 285].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [226]
Section 3.2 (p. 4) says the authors use human crowdworkers to assess the factual correctness of responses from neural dialogue generators. This supports human evaluation as an approach, but not the broader claim that chatbots are usually evaluated this way; the Introduction (p. 2) describes human-annotated datasets for open-domain dialogue as very limited. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228 · see citing works
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., D., M., Châtel, G., Chaumond, J., Rault, T., Voss, C., Wang, F., et al. (2019) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.3385997 · Cited by 1,054 · see citing works
Recipes for Building an Open-Domain Chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., et al. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.24 · Cited by 784 · see citing works
Self-citationEvaluating the Factual Consistency of Abstractive Text Summarization
Kryscinski, W., McCann, B., Xiong, C., & Socher, R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.750 · Cited by 669 · see citing works
Retrieval Augmentation Reduces Hallucination in Conversation
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.320 · Cited by 575 · see citing works
Commonsense Knowledge Aware Conversation Generation with Graph Attention
Zhou, H., Young, T., Huang, M., Zhao, H., Xu, J., & Zhu, X. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/643 · Cited by 495 · see citing works
A Knowledge-Grounded Neural Conversation Model
Ghazvininejad, M., Brockett, C., Chang, M.-W., Dolan, B., Gao, J., Yih, W.-T., & Galley, M. (2018) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v32i1.11977 · Cited by 457 · see citing works
TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational AgentsPreprint
Wolf, T., Sanh, V., Chaumond, J., & Delangue, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08149 · Cited by 435 · see citing works
KILT: a Benchmark for Knowledge Intensive Language Tasks
Petroni, F., Piktus, A., Fan, A., Lewis, P., Yazdani, M., De Cao, N., Thorne, J., Jernite, Y., Karpukhin, V., Maillard, J., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.200 · Cited by 357 · see citing works
Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Schuster, T., Fisch, A., & Barzilay, R. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.52 · Cited by 126 · see citing works
Detecting Hallucinated Content in Conditional Neural Sequence Generation
Zhou, C., Neubig, G., Gu, J., Diab, M., Guzmán, F., Zettlemoyer, L., & Ghazvininejad, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.120 · Cited by 122 · see citing works
Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., & Abend, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.619 · Cited by 113 · cited 3× · see citing works
“While factual correctness has been studied in knowledge-grounded response generation (Li et al., 2019; Shuster et al., 2021; Honovich et al., 2021; Rashkin et al., 2021), there has been very limited work in creating a human annotated dataset and detecting factual consistency for open-domain dialog…”
BackgroundIncremental Transformer with Deliberation Decoder for Document Grounded Conversations
Li, Z., Niu, C., Meng, F., Feng, Y., Li, Q., & Zhou, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1002 · Cited by 109 · see citing works
Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding
Dziri, N., Madotto, A., Zaïane, O., & Bose, A. J. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.168 · Cited by 81 · see citing works
A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems
Thomson, C., & Reiter, E. (2020) · Proceedings of the 13th International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.22 · Cited by 38 · see citing works
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions
Roller, S., Boureau, Y.-L., Weston, J., Bordes, A., Dinan, E., Fan, A., Gunning, D., Ju, D., Li, M., Poff, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2006.12442 · Cited by 38 · see citing works
Truth or Error? Towards systematic analysis of factual errors in abstractive summaries
Lux, K.-M., Sappelli, M., & Larson, M. (2020) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2020.eval4nlp-1.1 · Cited by 14 · see citing works
Improving Factual Consistency of Abstractive Summarization via Question AnsweringPreprint
Nan, F., dos Santos, C. N., Zhu, H., Ng, P., McKeown, K. R., Nallapati, R. M., Zhang, D., Wang, Z., Arnold, A. O., & Xiang, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.04623 · Cited by 4 · see citing works
Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable FeaturesPreprint
Rashkin, H., Reitter, D., Tomar, G. S., & Das, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.06963 · Cited by 1 · see citing works
Flagged first, then by citations
1 of the 25 references couldn’t be matched to a known paper.
Research works printed but not linked: 16
- Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net.
- Ondˇ rej Dušek and Zdenˇ ek Kasner. 2020. Evaluating semantic accuracy of data-to-text generation with natural language inference. In Proceedings of the 13th International Conference on Natural Language Generation , pages 131-137, Dublin, Ireland. Association for Computational Linguistics.
- Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek HakkaniTür. 2019. Topical-chat: Towards knowledgegrounded open-domain conversations. Proc. Interspeech 2019 , pages 1891-1895.
- Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net.
- Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 67696781, Online. Association for Computational Linguistics.
- Robert Logan, Nelson F. Liu, Matthew E. Peters, Matt Gardner, and Sameer Singh. 2019. Barack's wife hillary: Using knowledge graphs for fact-aware language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 5962-5971, Florence, Italy. Association for Computational Linguistics.
- Luca Massarelli, Fabio Petroni, Aleksandra Piktus, Myle Ott, Tim Rocktäschel, Vassilis Plachouras, Fabrizio Silvestri, and Sebastian Riedel. 2020. How decoding strategies affect the verifiability of generated text. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 223-235, Online. Association for Computational Linguistics.
- Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 1906-1919, Online. Association for Computational Linguistics.
- Sabrina J Mielke, Arthur Szlam, Y-Lan Boureau, and Emily Dinan. 2020. Linguistic calibration through metacognition: aligning dialogue agent responses with expected correctness. arXiv preprint arXiv:2012.14983 . · arXiv
- Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP) , pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9.
- Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
- Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 5418-5426, Online. Association for Computational Linguistics.
- Tal Schuster, Adam Fisch, and Regina Barzilay. 2021b. Get your vitamin c! robust fact verification with contrastive evidence. arXiv preprint arXiv:2103.08541 . · arXiv
- James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long
- Semih Yavuz, Abhinav Rastogi, Guan-Lin Chao, and Dilek Hakkani-Tur. 2019. DeepCopy: Grounded response generation with hierarchical pointer networks. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue , pages 122-132, Stockholm, Sweden. Association for Computational Linguistics.
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
“Therefore, chatbots are usually evaluated by humans on factual consistency or factual correctness [154, 198].”
BackgroundMethodInfluentialHuang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516 · see citing works
Measuring Attribution in Natural Language Generation Models
Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2023) · Computational Linguistics · doi:10.1162/coli_a_00486 · Cited by 71 · see citing works
FaithDial : A Faithful Benchmark for Information-Seeking Dialogue
Dziri, N., Kamalloo, E., Milton, S., Zai͏̈ane, O. R., Yu, M., Ponti, E. M., & Reddy, S. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00529 · Cited by 68 · see citing works
Kim, S. S. Y., Vaughan, J. W., Liao, Q. V., Lombrozo, T., & Russakovsky, O. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714020 · Cited by 58 · see citing works
“However, the responses produced by LLMs are often inaccurate, sometimes in subtle ways [27, 51, 103, 108].”
Self-citationUnderstanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection
Xu, W., Agrawal, S., Briakou, E., Martindale, M. J., & Carpuat, M. J. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00563 · Cited by 59 · see citing works
Open-domain Dialogue Generation: What We Can Do, Cannot Do, And Should Do Next
Kann, K., Ebrahimi, A., Koh, J. J., Dudy, S., & Roncone, A. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.13 · Cited by 25 · see citing works
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Song, D., Xie, X., Song, J., Zhu, D., Huang, Y., Juefei-Xu, F., & Ma, L. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3411928 · Cited by 25 · see citing works
Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identity
Shuster, K., Urbanek, J., Szlam, A., & Weston, J. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.182 · Cited by 20 · see citing works
Controllable Mixed-Initiative Dialogue Generation through Prompting
Chen, M., Yu, X., Shi, W., Awasthi, U., & Zhou, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.82 · Cited by 23 · see citing works
RuCoLA: Russian Corpus of Linguistic Acceptability
Mikhailov, V., Shamardina, T., Ryabinin, M., Pestova, A., Smurov, I., & Artemova, E. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.348 · Cited by 21 · see citing works
Measuring Attribution in Natural Language Generation Models
Rashkin, H., Nikolaev, V. A., Lamm, M., Aroyo, L. M., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2023) · Computational Linguistics · doi:10.1162/coli_a_00490 · Cited by 16 · see citing works
Don’t Forget Your ABC’s: Evaluating the State-of-the-Art in Chat-Oriented Dialogue Systems
Finch, S. E., Finch, J. D., & Choi, J. D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.839 · Cited by 10 · see citing works
A Brief Survey on Safety of Large Language Models
Gao, Z., Liu, X., Lan, Y.-S., & Yang, Z. (2024) · Journal of Computing and Information Technology · doi:10.20532/cit.2024.1005778 · Cited by 10 · see citing works
Xue, B., Wang, W., Wang, H., Mi, F., Wang, R., Wang, Y., Shang, L., Jiang, X., Liu, Q., & Wong, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.525 · Cited by 11 · see citing works
Detecting Dialogue Hallucination Using Graph Neural Networks
Furumai, K., Wang, Y., Shinohara, M., Ikeda, K., Yu, Y., & Kato, T. (2023) · International Conference on Machine Learning and Applications · doi:10.1109/icmla58977.2023.00128 · Cited by 5 · see citing works
Eyes Show the Way: Modelling Gaze Behaviour for Hallucination Detection
Maharaj, K., Saxena, A., Kumar, R., Mishra, A., & Bhattacharyya, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.764 · Cited by 4 · see citing works
Nandwani, Y., Kumar, V., Raghu, D., Joshi, S., & Lastras, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.639 · Cited by 5 · see citing works
Towards In-Context Non-Expert Evaluation of Reflection Generation for Counselling Conversations
Wu, Z., Balloccu, S., Helaoui, R., Recupero, D. R., & Riboni, D. (2022) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2022.gem-1.9 · Cited by 2 · see citing works
Factual or Contextual? Disentangling Error Types in Entity Description Generation
Goyal, N., Nenkova, A., & Daumé, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.463 · Cited by 1 · see citing works
Umigon-lexicon: rule-based model for interpretable sentiment analysis and factuality categorization
Levallois, C. (2024) · Language Resources and Evaluation · doi:10.1007/s10579-024-09742-y · Cited by 1 · see citing works
Controllable Factuality in Document-Grounded Dialog Systems Using a Noisy Channel Model
Daheim, N., Thulke, D., Dugast, C., & Ney, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.98 · Cited by 1 · see citing works
Image Captioning with External Knowledge
Nikiforova, S. (2023) · doi:10.33540/1709
- Authors
- Santhanam, S., Hedayatnia, B., Gella, S., Padmakumar, A., Kim, S., Liu, Y., & Hakkani-Tur, D.
- Affiliations
- University of North Carolina at Charlotte (US)
- Amazon (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2110.05456
- OpenAlex
- W3203390572
- Cited by
- 21
- References
- 25
- Citations by year
- 2024: 3 · 2023: 11 · 2022: 7
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- factual accuracy, hallucination detection, factual consistency, manual annotation, decoding strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
