The DOI matches its Crossref record; 41 of the 41 references it cites were found in a registry or an index.
LM vs LM: Detecting Factual Errors via Cross Examination
Open accessAbstract
A prominent weakness of modern language models (LMs) is their tendency to generate factually incorrect text, which hinders their usability.A natural question is whether such factual errors can be detected automatically.Inspired by truth-seeking mechanisms in law, we propose a factuality evaluation framework for LMs that is based on cross-examination.Our key idea is that an incorrect claim is likely to result in inconsistency with other claims that the model generates.To discover such inconsistencies, we facilitate a multi-turn interaction between the LM that generated the claim and another LM (acting as an examiner) which introduces questions to discover inconsistencies.We empirically evaluate our method on factual claims made by multiple recent LMs on four benchmarks, finding that it outperforms existing methods and baselines, often by a large gap.Our results demonstrate the potential of using interacting LMs to capture factual errors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Cohen et al. introduced the LMvLM approach, which leverages an examiner LM to question an examinee LM, aiming to unveil inconsistencies of claims during multi-turn interaction.
“Drawing inspiration from legal cross-examination practices, Cohen et al. [62] introduced the LMvLM approach. This strategy leverages an examiner LM to question an examinee LM, aiming to unveil inconsistencies of claims during multi-turn interaction.”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [62]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.173 · Cited by 1,185
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
How Can We Know What Language Models Know?
Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00324 · Cited by 1,002
Kuhn, L., Gal, Y., & Farquhar, S. (2023) · arXiv · doi:10.48550/arxiv.2302.09664 · Cited by 700
Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.546 · Cited by 340
Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims
Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07213 · Cited by 306
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., & Lewis, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.378 · Cited by 303
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2210.03350 · Cited by 294
Generalization through Memorization: Nearest Neighbor Language Models
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00172 · Cited by 285
Factual Probing Is [MASK]: Learning vs. Learning to Recall
Zhong, Z., Friedman, D., & Chen, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.398 · Cited by 274
Dialogue Natural Language Inference
Welleck, S., Weston, J., Szlam, A., & Cho, K. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1363 · Cited by 249
Locating and Editing Factual Associations in GPT
Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022) · arXiv · doi:10.48550/arxiv.2202.05262 · Cited by 230
Measuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220
The Fact Extraction and VERification (FEVER) Shared Task
Thorne, J., Vlachos, A., Cocarascu, O., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the First Workshop on Fact Extraction and VERification (FEVER) · doi:10.18653/v1/w18-5501 · Cited by 212
Calibration of Pre-trained Transformers
Desai, S., & Durrett, G. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.21 · Cited by 209
Language Models (Mostly) Know What They Know
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05221 · Cited by 187
Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00407 · Cited by 177
Evaluating Verifiability in Generative Search Engines
Liu, N., Zhang, T., & Liang, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.467 · Cited by 143
RARR: Researching and revising what language models say, using language models
Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A. T., Fan, Y., Zhao, V., Lao, N., Lee, H., Juan, D.-C., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.910 · Cited by 141
Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., & Abend, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.619 · Cited by 113
Irving, G., Christiano, P., & Amodei, D. (2018) · arXiv · doi:10.48550/arxiv.1805.00899 · Cited by 71
TRUE: Re-evaluating Factual Consistency Evaluation
Honovich, O., Aharoni, R., Herzig, J., Taitelbaum, H., Kukliansy, D., Cohen, V., Scialom, T., Szpektor, I., Hassidim, A., & Matias, Y. (2022) · Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering · doi:10.18653/v1/2022.dialdoc-1.19 · Cited by 56
PEER: A Collaborative Language Model
Schick, T., Dwivedi-Yu, J., Jiang, Z., Petroni, F., Lewis, P., Izacard, G., You, Q., Nalmpantis, C., Grave, E., & Riedel, S. (2022) · arXiv · doi:10.48550/arxiv.2208.11663 · Cited by 49
Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation
Bulian, J., Buck, C., Gajewski, W., Börschinger, B., & Schuster, T. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.20 · Cited by 44
Evaluating Factuality in Text Simplification
Devaraj, A., Sheffield, W., Wallace, B., & Li, J. J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.506 · Cited by 40
Answering Questions by Meta-Reasoning over Multiple Chains of Thought
Yoran, O., Wolfson, T., Bogin, B., Katz, U., Deutch, D., & Berant, J. (2023) · arXiv · doi:10.48550/arxiv.2304.13007 · Cited by 38
Measuring Attribution in Natural Language Generation Models
Rashkin, H., Nikolaev, V., Lamm, M., Aroyo, L., Collins, M., Das, D., Petrov, S., Tomar, G. S., Turc, I., & Reitter, D. (2021) · arXiv · doi:10.48550/arxiv.2112.12870 · Cited by 34
TrueTeacher: Learning Factual Consistency Evaluation with Large Language Models
Gekhman, Z., Herzig, J., Aharoni, R., Elkind, C., & Szpektor, I. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.127 · Cited by 33
Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
Bohnet, B., Tran, V. Q., Verga, P., Aharoni, R., Andor, D., Soares, L. B., Ciaramita, M., Eisenstein, J., Ganchev, K., Herzig, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.08037 · Cited by 30
Flagged first, then by citations
Research works printed but not linked: 34
- Dimitrios Alivanistos, Selene Báez Santamaría, Michael Cochez, Jan-Christoph Kalo, Emile van Krieken, and Thiviyan Thanapalasingam. 2022. Prompting as probing: Using language models for knowledge base construction. arXiv preprint arXiv:2208.11057. · arXiv
- Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565. · arXiv
- Pepa Atanasova, Jakob Grue Simonsen, Christina Li- oma, and Isabelle Augenstein. 2022. Fact checking with insufficient evidence. Transactions of the Asso- ciation for Computational Linguistics, 10:746-763. · DOI
- Sebastian Borgeaud, Arthur Mensch, Jordan Hoff- mann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from tril- lions of tokens. In International conference on ma- chine learning, pages 2206-2240. PMLR.
- Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. 2023. Crawling the internal knowledge- base of language models. In Findings of the Asso- ciation for Computational Linguistics: EACL 2023, pages 1856-1869, Dubrovnik, Croatia. Association for Computational Linguistics.
- Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2022. Evaluating attribution in dialogue sys- tems: The BEGIN benchmark. Transactions of the Association for Computational Linguistics, 10:1066- 1083. · DOI
- Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. 2023. The capacity for moral self- correction in large language models. arXiv preprint arXiv:2302.07459. · arXiv
- Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. 2023. Trueteacher: Learn- ing factual consistency evaluation with large lan- guage models. arXiv preprint arXiv:2305.11171. · arXiv
- Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International confer- ence on machine learning, pages 3929-3938.
- PMLR.
- Shibo Hao, Bowen Tan, Kaiwen Tang, Hengzhe Zhang, Eric P Xing, and Zhiting Hu. 2022. Bertnet: Har- vesting knowledge graphs from pretrained language models. arXiv preprint arXiv:2206.14268. · arXiv
- Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. 2021. Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916. · arXiv
- Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tomp- son, Igor Mordatch, Yevgen Chebotar, et al. 2022. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608. · arXiv
- Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brah- man, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022. Maieutic prompting: Logically consistent reasoning with recursive explanations. In Conference on Empirical Methods in Natural Lan- guage Processing.
- Amita Kamath, Robin Jia, and Percy Liang. 2020. Se- lective question answering under domain shift. arXiv preprint arXiv:2006.09462. · arXiv
- Ryo Kamoi, Tanya Goyal, and Greg Durrett. 2023. Shortcomings of question answering based factuality frameworks for error localization. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 132-146, Dubrovnik, Croatia. Association for Com- putational Linguistics.
- Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334. · arXiv
- Jieyi Long. 2023. Large language model guided tree-of- thought. arXiv preprint arXiv:2305.08291. · arXiv
- Benjamin Newman, Prafulla Kumar Choubey, and Nazneen Rajani. 2021. P-adapters: Robustly extract- ing factual information from language models with diverse prompts. arXiv preprint arXiv:2110.07280. · arXiv
- Joon Sung Park, Lindsay Popowski, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2022. Social simulacra: Creating popu- lated prototypes for social computing systems. In The 35th Annual ACM Symposium on User Inter- face Software and Technology, UIST 2022, Bend, OR, USA, 29 October 2022 -2 November 2022, pages 74:1-74:18. ACM. · DOI · Paper page
- Fabio Petroni, Tim Rocktäschel, Patrick Lewis, An- ton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowl- edge bases? arXiv preprint arXiv:1909.01066. · arXiv
- Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5203-5212, Online. Association for Computa- tional Linguistics. · DOI · Paper page
- Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, Alex Wang, and Patrick Gallinari. 2021. QuestEval: Summariza- tion asks for fact-based evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 6594-6604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. · DOI · Paper page
- Meiqi Sun, Wilson Yan, Pieter Abbeel, and Igor Mor- datch. 2022. Quantifying uncertainty in foundation models via ensembles. In NeurIPS 2022 Workshop on Robustness in Sequence Modeling.
- Derek Tam, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan, Mohit Bansal, and Colin Raffel. 2022. Eval- uating the factual consistency of large language models through summarization. arXiv preprint arXiv:2211.08412. · arXiv
- Neeraj Varshney, Swaroop Mishra, and Chitta Baral. 2022. Investigating selective prediction approaches across several tasks in IID, OOD, and adversarial settings. In Findings of the Association for Compu- tational Linguistics: ACL 2022, pages 1995-2002, Dublin, Ireland. Association for Computational Lin- guistics. · DOI
- Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the fac- tual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 5008-5020, Online. Asso- ciation for Computational Linguistics. · DOI
- Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022. Self-consistency im- proves chain of thought reasoning in language mod- els. arXiv preprint arXiv:2203.11171. · arXiv
- Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. · arXiv
- Dustin Wright, David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Isabelle Augenstein, and Lucy Lu Wang. 2022. Generating scientific claims for zero- shot scientific fact checking. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 2448-2460, Dublin, Ireland. Association for Compu- tational Linguistics. · DOI
- Hiyori Yoshikawa and Naoaki Okazaki. 2023. Selective- LAMA: Selective prediction for confidence-aware evaluation of language models. In Findings of the As- sociation for Computational Linguistics: EACL 2023, pages 2017-2028, Dubrovnik, Croatia. Association for Computational Linguistics.
- Polina Zablotskaia, Du Phan, Joshua Maynez, Shashi Narayan, Jie Ren, and Jeremiah Liu. 2023. On un- certainty calibration and selective generation in prob- abilistic neural summarization: A benchmark study. arXiv preprint arXiv:2304.08653. · arXiv
- Zexuan Zhong, Tao Lei, and Danqi Chen. 2022. Train- ing language models with memory augmentation. In Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing, pages 5657-5673, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
- Chunting Zhou, Junxian He, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2022. Prompt consistency for zero-shot task generalization. arXiv preprint arXiv:2205.00049. · arXiv
Cited by
Sorted by citations
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
Factuality challenges in the era of large language models and opportunities for fact-checking
Augenstein, I., Baldwin, T., Cha, M., Chakraborty, T., Ciampaglia, G. L., Corney, D., DiResta, R., Ferrara, E., Hale, S., Halevy, A., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00881-z · Cited by 187
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141
On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
Stechly, K., Valmeekam, K., & Kambhampati, S. (2024) · arXiv · doi:10.48550/arxiv.2402.08115 · Cited by 120
Can Large Language Models Really Improve by Self-critiquing Their Own Plans?
Valmeekam, K., Marquez, M., & Kambhampati, S. (2023) · arXiv · doi:10.48550/arxiv.2310.08118 · Cited by 90
When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs
Kamoi, R., Zhang, Y., Zhang, N., Han, J., & Zhang, R. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00713 · Cited by 200
Benchmarking Cognitive Biases in Large Language Models as Evaluators
Koo, R., Lee, M., Raheja, V., Park, J. I., Kim, Z. M., & Kang, D. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.29 · Cited by 70
Scaling Large Language Model-based Multi-Agent Collaboration
Qian, C., Xie, Z., Wang, Y., Liu, W., Zhu, K., Xia, H., Dang, Y., Du, Z., Chen, W., Yang, C., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.07155 · Cited by 8
Measuring Collapse and Correction in Homogeneous-Panel LLM Debate
Li, X., Liu, M., & Yuen, C. (2026) · arXiv · doi:10.48550/arxiv.2609.35279
Metadata
- Authors
- Cohen, R., Hamri, M., Geva, M., & Globerson, A.
- Affiliations
- Tel Aviv University (IL)
- Google DeepMind (United Kingdom) (GB)
- Google (United States) (US)
- Corresponding author
- Hamri, May
- ORCID
- Venue
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Published
- 2023
- Pages
- 12621-12640
- OpenAlex
- W4389520333
- Cited by
- 60
- References
- 41
- Citations by year
- 2026: 20 · 2025: 20 · 2024: 16 · 2023: 4
- FWCI
- 10.63
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, cross-examination, factuality evaluation, multi-turn interaction, inconsistency detection, factual claims
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
