Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence
Open accessAbstract
Question answering models can use rich knowledge sources -- up to one hundred retrieved passages and parametric knowledge in the large-scale language model (LM). Prior work assumes information in such knowledge sources is consistent with each other, paying little attention to how models blend information stored in their LM parameters with that from retrieved evidence documents. In this paper, we simulate knowledge conflicts (i.e., where parametric knowledge suggests one answer and different passages suggest different answers) and examine model behaviors. We find retrieval performance heavily impacts which sources models rely on, and current models mostly rely on non-parametric knowledge in their best-performing settings. We discover a troubling trend that contradictions among knowledge sources affect model confidence only marginally. To address this issue, we present a new calibration study, where models are discouraged from presenting any single answer when presented with multiple conflicting answer candidates in retrieved evidences.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Retrieval-based methods face key challenges: reliable resolution of conflicts between parametric and retrieved knowledge, and maintaining improvements in factuality as model size increases.
“retrieval-based methods face key challenges, namely reliable resolution of conflicts between parametric and retrieved knowledge (Longpre et al., 2022; Chen et al., 2022) as well as maintaining improvements in factuality as model size increases (Mallen et al., 2023).”
From Fine-tuning Language Models for Factuality (2023) · cites this paper as Chen et al., 2022
Cites
XGBoost: A Scalable Tree Boosting System
Chen, T., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939785 · Cited by 54,569 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682 · see citing works
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575 · see citing works
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
Bidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01603 · Cited by 2,322 · see citing works
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834 · see citing works
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · see citing works
Self-citationReading Wikipedia to answer open-domain questions
Chen, D., Fisch, A., Weston, J., & Bordes, A. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1171 · Cited by 1,546 · see citing works
Adversarial Examples for Evaluating Reading Comprehension Systems
Jia, R., & Liang, P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1215 · Cited by 1,410 · see citing works
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.74 · Cited by 1,109 · see citing works
Beyond accuracy: behavioral testing of NLP models with CheckList
Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.442 · Cited by 890 · see citing works
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · see citing works
Latent retrieval for weakly supervised open domain question answering
Lee, K., Chang, M.-W., & Toutanova, K. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1612 · Cited by 771 · see citing works
How Much Knowledge Can You Pack into the Parameters of a Language Model?
Roberts, A., Raffel, C., & Shazeer, N. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.437 · Cited by 638 · see citing works
REALM: Retrieval-Augmented Language Model Pre-Training
Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020) · arXiv · doi:10.48550/arxiv.2002.08909 · Cited by 574 · see citing works
Learning the Difference that Makes a Difference with Counterfactually-Augmented DataPreprint
Kaushik, D., Hovy, E. H., & Lipton, Z. C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.12434 · Cited by 346 · see citing works
ELI5: long form question answering
Fan, A., Jernite, Y., Perez, E., Grangier, D., Weston, J., & Auli, M. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1346 · Cited by 319 · see citing works
Universal Dependency Parsing from Scratch
Qi, P., Dozat, T., Zhang, Y., & Manning, C. D. (2018) · Proceedings of the · doi:10.18653/v1/k18-2016 · Cited by 268 · see citing works
Evaluating Models’ Local Decision Boundaries via Contrast Sets
Gardner, M., Artzi, Y., Basmov, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.117 · Cited by 265 · see citing works
Scaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · see citing works
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., & Chen, D. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5801 · Cited by 220 · see citing works
Self-citationAmbigQA: Answering Ambiguous Open-domain Questions
Min, S., Michael, J., Hajishirzi, H., & Zettlemoyer, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.466 · Cited by 219 · see citing works
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models
Wu, T., Ribeiro, M. T., Heer, J., & Weld, D. S. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.523 · Cited by 210 · see citing works
Selective Question Answering under Domain Shift
Kamath, A., Jia, R., & Liang, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.503 · Cited by 153 · see citing works
Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasets
Lewis, P., Stenetorp, P., & Riedel, S. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.86 · Cited by 145 · see citing works
Enabling Language Models to Fill in the Blanks
Donahue, C., Lee, M., & Liang, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.225 · Cited by 139 · see citing works
Hurdles to Progress in Long-form Question Answering
Krishna, K., Roy, A., & Iyyer, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.393 · Cited by 114 · see citing works
Seeing Things from a Different Angle:Discovering Diverse Perspectives about Claims
Chen, S., Khashabi, D., Yin, W., Callison-Burch, C., & Roth, D. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1053 · Cited by 104 · see citing works
Beat the ai: Investigating adversarial human annotation for reading comprehension
Bartolo, M., Roberts, A., Welbl, J., Riedel, S., & Stenetorp, P. (2020) · UCL Discovery (University College London) · Cited by 102 · see citing works
Distilling Knowledge from Reader to Retriever for Question Answering
Izacard, G., & Grave, E. (2020) · arXiv · doi:10.48550/arxiv.2012.04584 · Cited by 102 · see citing works
Entity-Based Knowledge Conflicts in Question Answering
Longpre, S., Perisetla, K., Chen, A., Ramesh, N., DuBois, C., & Singh, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.565 · Cited by 101 · see citing works
Is Retriever Merely an Approximator of Reader?Preprint
Yang, S., & Seo, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10999 · Cited by 33 · see citing works
Synthetic Disinformation Attacks on Automated Fact Verification Systems
Du, Y., Bosselut, A., & Manning, C. D. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21302 · Cited by 27 · see citing works
Retrieval-guided Counterfactual Generation for QA
Paranjape, B., Lamm, M., & Tenney, I. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.117 · Cited by 19 · see citing works
Attacking Open-domain Question Answering by Injecting MisinformationPreprint
Pan, L., Chen, W., Kan, M., & Wang, W. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07803 · Cited by 3 · see citing works
Automatic Fact-Guided Sentence Modification
Shah, D., Schuster, T., & Barzilay, R. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6406 · Cited by 2 · see citing works
Reasoning over Logically Interacted Conditions for Question AnsweringPreprint
Sun, H., Cohen, W. W., & Salakhutdinov, R. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.12898 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.546 · Cited by 340 · see citing works
Hallucination Mitigation for Retrieval-Augmented Large Language Models: A Review
Zhang, W., & Zhang, J. (2025) · Mathematics · doi:10.3390/math13050856 · Cited by 145 · see citing works
On the Risk of Misinformation Pollution with Large Language Models
Pan, Y., Pan, L., Chen, W., Nakov, P., Kan, M.-Y., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.97 · Cited by 70 · see citing works
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances
Zhang, Z., Fang, M., Chen, L., Namazi‐Rad, M., & Wang, Y.-X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.516 · Cited by 37 · see citing works
DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering
Neeman, E., Aharoni, R., Honovich, O., Choshen, L., Szpektor, I., & Abend, O. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.559 · Cited by 41 · see citing works
A Causal View of Entity Bias in (Large) Language Models
Wang, F., Mo, W., Wang, Y., Zhou, W., & Chen, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1013 · Cited by 18 · see citing works
Fine-tuning Language Models for Factuality
Tian, K., Mitchell, E., Yao, H., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2311.08401 · Cited by 14 · see citing works
“…to verify consistency; however, retrieval-based methods face key challenges, namely reliable resolution of conflicts between parametric and retrieved knowledge (Longpre et al., 2022; Chen et al., 2022) as well as maintaining improvements in factuality as model size increases (Mallen et al., 2023).”
Attacking Open-domain Question Answering by Injecting Misinformation
Pan, L., Chen, W., Kan, M.-Y., & Wang, W. Y. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.35 · Cited by 9 · see citing works
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding
Shi, W., Han, X., Lewis, M., Tsvetkov, Y., Zettlemoyer, L., & Yih, S. W.-T. (2023) · arXiv · doi:10.48550/arxiv.2305.14739 · Cited by 6 · see citing works
“However, simply adding documents does not always change the model predictions, as current LMs often overlook the contexts and rely heavily on their prior parametric knowledge (Longpre et al., 2021; Chen et al., 2022).”
BackgroundMitigating Temporal Misalignment by Discarding Outdated Facts
Zhang, M., & Choi, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.879 · Cited by 8 · see citing works
Self-citationICR: A framework for resolving knowledge conflicts in retrieval-augmented generation
Xiong, Y., Chen, Y., & Zhang, H. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.132139 · Cited by 6 · see citing works
Penny‐Dimri, J. C., Bachmann, M., Cooke, W. R., Mathewlynn, S., Dockree, S., Tolladay, J., Kossen, J., Li, L., Gal, Y., & Jones, G. D. (2025) · The Lancet Obstetrics Gynaecology & Women s Health · doi:10.1016/j.lanogw.2025.100005 · Cited by 4 · see citing works
KGQuiz: Evaluating the Generalization of Encoded Knowledge in Large Language Models
Bai, Y., Feng, S., Balachandran, V., Tan, Z., Lou, S., He, T., & Tsvetkov, Y. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645623 · Cited by 5 · see citing works
Park, S.-I., & Lee, J.-Y. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00724 · Cited by 5 · see citing works
LedgerRAG: Governance-Driven Agentic Chain of Retrieval for Dynamic Knowledge Scenarios
Wang, S., Zhang, Y., Guo, Y., & Kang, J. (2026) · Electronics · doi:10.3390/electronics15071376 · Cited by 2 · see citing works
ConSens: Assessing Context Grounding in Open-Book Question Answering
Vankov, I., Ivanov, M., Correia, A. D., & Botev, V. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-04549-2_13 · Cited by 1 · see citing works
Nguyen, V. B., Seifert, C., & Schlötterer, J. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-08330-2_8 · Cited by 1 · see citing works
Wu, H., Wang, C., Sun, C., Lu, C., Zhang, Z., & Chen, Y. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792289 · Cited by 1 · see citing works
Mitigating Knowledge Conflicts in Data-to-Text Generation via the Internalization of Fact Extraction
Mo, X., Xiang, Y., Pan, Y., Hou, Y., & Luo, P. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651167 · Cited by 1 · see citing works
Dissecting Dissonance: Benchmarking Large Multimodal Models Against Self-Contradictory Instructions
Gao, J., Gan, L., Li, Y., Ye, Y., & Wang, D. Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72998-0_23 · Cited by 1 · see citing works
SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models
Pham, T., Nguyen, N., Zunjare, P., Chen, W., Tseng, Y.-M., & Vu, T. (2025) · arXiv · doi:10.48550/arxiv.2506.01062
“Models also exhibit confirmation bias by favoring evidence that aligns with their parametric memory [Chen et al., 2022], often resolving contradictions in favor of internal knowledge [Jin et al., 2024, Jiayang et al., 2024].”
BordaRAG: Resolving Knowledge Conflict in Retrieval-Augmented Generation via Borda Voting Process
Li, Y., Xu, C., Xu, J., & Wen, J.-R. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761038
Eliminating Retrieval Knowledge Conflicts: Cross-Validation Re-ranking with Large Language Models
Wu, Q., Lin, H., Zheng, H., Xie, R., Yang, S., Sun, X., Kang, Z., & Kim, H. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228012
Enhancing User Alignment in Large Language Models via Counterfactual Context Divergence DecodingPreprint
Wang, Y., Wang, J., & Luo, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5644037
Mitigating Context-Memory Conflicts in LLMs Through Dynamic Cognitive Reconciliation Decoding
Zhou, Y., Li, W., Lu, Y., Wang, Y., Liu, X., Wang, W., Yu, J., Zhang, M., & Li, J. (2026) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2026.3657980
Self-citationKnowFC: Navigating Knowledge Conflicts in Large Language Model-based Fact-Checking
Zhang, Y., Zhou, S., Li, X., Tian, Z., Gao, Y., Zhang, S., Hou, W., Liu, Y., & Zhou, B. (2026) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3773966.3777935
When Retrieval Hurts: A Critical Analysis of RAG in Medical Question Answering
Xie, W., Fang, Y., Zheng, X., Liu, D., & Li, Z. (2025) · International Conference on Artificial Intelligence, Systems and Network Security (AISNS) · doi:10.1145/3797161.3797211
Hierarchical Voting Decoder for Resolving Knowledge Conflicts
An, L., Chen, S., Li, Z., Wu, K., Yang, S., Pei, X., Wang, J., & Liu, S. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461241
Mitigating Entity Bias in Relation Extraction with Pair-Training
Jia, H., Zhu, T., Tan, C., & Chen, W. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11464075
Lee, G., Kim, S., Byun, Y., Lee, J., & Choi, H. (2026) · International Journal of Precision Engineering and Manufacturing-Green Technology · doi:10.1007/s40684-026-00890-w · Cited by 2 · see citing works
Trust the Eye or the Crowd? Unveiling Conformity in Multimodal Large Language Models
Luo, H., Niu, Y., Liu, H., Sun, Y., Yang, Z., Chen, T., Liu, D., Miao, Y., 周景师, Zhang, H., et al. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11582657
Exploring Knowledge Conflicts for Faithful LLM Reasoning: Benchmark and Method
Zhao, T., Chen, J., Zhang, S., Zhu, H., Lin, Q., & Liu, J. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809650
- Authors
- Chen, H.-T., Zhang, M. J. Q., & Choi, E.
- Affiliations
- The University of Texas at Austin (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Pages
- 2292-2307
- arXiv
- 2210.13701
- OpenAlex
- W4385574183
- Cited by
- 31
- References
- 42
- Citations by year
- 2026: 9 · 2025: 9 · 2024: 6 · 2023: 7
- FWCI
- 3.04
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- University of Texas at Austin
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- question answering, knowledge conflicts, conflicting evidence, knowledge sources, retrieval-augmented generation, large language models, model calibration, model confidence, parametric knowledge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
