CoChat
Sign Up
PreprintCited by 2Network Security and Intrusion DetectionInformation and Cyber SecurityAdversarial Robustness in Machine Learning

Benchmarking LLMs in an Embodied Environment for Blue Team Threat Hunting

Open access
Liu, Xiaoqun, Yu, Feiyang, Li, Xi, Yan, Guanhua, Yang, Ping, Xi, Zhaohan·2025·arXiv
DOI 10.48550/arxiv.2505.11901arXiv 2505.11901
Open this paper in CoChat — free accountOpen PDF

Abstract

As cyber threats continue to grow in scale and sophistication, blue team defenders increasingly require advanced tools to proactively detect and mitigate risks. Large Language Models (LLMs) offer promising capabilities for enhancing threat analysis. However, their effectiveness in real-world blue team threat-hunting scenarios remains insufficiently explored. In this paper, we present CYBERTEAM, a benchmark designed to guide LLMs in blue teaming practice. CYBERTEAM constructs an embodied environment in two stages. First, it models realistic threat-hunting workflows by capturing the dependencies among analytical tasks from threat attribution to incident response. Next, each task is addressed through a set of embodied functions tailored to its specific analytical requirements. This transforms the overall threat-hunting process into a structured sequence of function-driven operations, where each node represents a discrete function and edges define the execution order. Guided by this framework, LLMs are directed to perform threat-hunting tasks through modular steps. Overall, CYBERTEAM integrates 30 tasks and 9 embodied functions, guiding LLMs through pipelined threat analysis. We evaluate leading LLMs and state-of-the-art cybersecurity agents, comparing CYBERTEAM's embodied function-calling against fundamental elicitation strategies. Our results offer valuable insights into the current capabilities and limitations of LLMs in threat hunting, laying the foundation for the practical adoption in real-world cybersecurity applications.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2505.11901

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing benchmarks largely focus on offensive workflows, with defensive behavior modeled as static rules or oracle responses, limiting their applicability for evaluating adversarial multi-agent dynamics such as coevolving red-team and blue-team agents.

    “Beyond their single-agent orientation, existing benchmarks largely focus on offensive workflows, with defensive behavior modeled as static rules or oracle responses, limiting their applicability for evaluating adversarial multi-agent dynamics such as coevolving red-team and blue-team agents adapting to each other over time (Ma et al., 2024;Liu et al., 2025).”

    From Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents (2025) · cites this paper as Liu et al., 2025

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 71

  1. [1][1] Tsedeke Abate, Kassa Michael, and Carl Angell. Assessment of scientific reasoning: Development and validation of scientific reasoning assessment tool. Eurasia Journal of Mathematics, Science and Technology Education , 16(12):em1927, 2020. -
  2. [2][2] Adel Abusitta, Miles Q Li, and Benjamin CM Fung. Malware classification and composition analysis: A survey of recent developments. Journal of Information Security and Applications , 59:102828, 2021. -
  3. [3][3] Ehsan Aghaei, Waseem Shadid, and Ehab Al-Shaer. Threatzoom: Cve2cwe using hierarchical neural network. arXiv preprint arXiv:2009.11501 , 2020. - · arXiv
  4. [5][5] Jamal Al-Karaki, Muhammad Al-Zafar Khan, and Marwan Omar. Exploring llms for malware detection: Review, framework design, and countermeasure approaches. arXiv preprint arXiv:2409.07587 , 2024. - · arXiv
  5. [6][6] Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, and Nidhi Rastogi. Ctibench: A benchmark for evaluating llms in cyber threat intelligence. arXiv preprint arXiv:2406.07599 , 2024. - · arXiv
  6. [8][8] Ali Alshumrani, Nathan Clarke, and Bogdan Ghita. A unified knowledge graph to permit interoperability of heterogenous digital evidence. In International Conference on Ubiquitous Security , pages 420-435. Springer, 2023. -
  7. [9][9] Mario Luca Bernardi, Marta Cimitile, and Riccardo Pecori. Automatic job safety report generation using rag-based llms. In 2024 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2024. -
  8. [10][10] Gavin Black, Varghese Vaidyan, and Gurcan Comert. Evaluating large language models for enhanced fuzzing: An analysis framework for llm-driven seed generation. IEEE Access , 2024. -
  9. [11][11] Andrea Borghesi, Federico Baldo, and Michela Milano. Improving deep learning models via constraint-based domain knowledge: a brief survey. arXiv preprint arXiv:2005.10691 , 2020. - · arXiv
  10. [12][12] Alexandre Braga, Ricardo Dahab, Nuno Antunes, Nuno Laranjeiro, and Marco Vieira. Practical evaluation of static analysis tools for cryptography: Benchmarking method and case study. In 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE) , pages 170-181. IEEE, 2017. -
  11. [13][13] Ioana Branescu, Octavian Grigorescu, and Mihai Dascalu. Automated mapping of common vulnerabilities and exposures to mitre att&ck tactics. Information , 15(4):214, 2024. -
  12. [14][14] Lee Brotherston, Amanda Berlin, and William F Reyor III. Defensive security handbook . " O'Reilly Media, Inc.", 2024. -
  13. [15][15] Sergio Caltagirone, Andrew Pendergast, and Christopher Betz. The diamond model of intrusion analysis. Threat Connect , 298(0704):1-61, 2013. -
  14. [16][16] Zhili Cheng, Yuge Tu, Ran Li, Shiqi Dai, Jinyi Hu, Shengding Hu, Jiahao Li, Yang Shi, Tianyu Yu, Weize Chen, et al. Embodiedeval: Evaluate multimodal llms as embodied agents. arXiv preprint arXiv:2501.11858 , 2025. - · arXiv
  15. [17][17] Leshem Choshen, Ariel Gera, Yotam Perlitz, Michal Shmueli-Scheuer, and Gabriel Stanovsky. Navigating the modern evaluation landscape: Considerations in benchmarks and frameworks for large language models (llms). In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024): Tutorial Summaries , pages 19-25, 2024. -
  16. [19][19] Ondrej ˇ Cupka, Ester Federlova, and Peter Vesely. Comparison of methodologies used in cybersecurity reports. In Developments in Information and Knowledge Management Systems for Business Applications: Volume 7 , pages 313-348. Springer, 2023. -
  17. [20][20] Hafsa Shareef Dar, M Ikramullah Lali, Moin Ul Din, Khalid Mahmood Malik, and Syed Ahmad Chan Bukhari. Frameworks for querying databases using natural language: a literature review. arXiv preprint arXiv:1909.01822 , 2019. - · arXiv
  18. [21][21] Tirtharaj Dash, Sharad Chitlangia, Aditya Ahuja, and Ashwin Srinivasan. A review of some techniques for inclusion of domain-knowledge into deep neural networks. Scientific Reports , 12(1):1040, 2022. -
  19. [22][22] Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. Pentestgpt: An llm-empowered automatic penetration testing tool. arXiv preprint arXiv:2308.06782 , 2023. - · arXiv
  20. [23][23] Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. { PentestGPT } : Evaluating and harnessing large language models for automated penetration testing. In 33rd USENIX Security Symposium (USENIX Security 24) , pages 847-864, 2024. -
  21. [24][24] Dharani Devadiga, Gordon Jin, Bisti Potdar, Hankyu Koo, Andrew Han, Anusha Shringi, Angad Singh, Kinjal Chaudhari, and Saurav Kumar. Gleam: Gan and llm for evasive adversarial malware. In 2023 14th International Conference on Information and Communication Technology Convergence (ICTC) , pages 53-58. IEEE, 2023. -
  22. [26][26] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint arXiv:2301.00234 , 2022. - · arXiv
  23. [27][27] Vardhan Dongre, Xiaocheng Yang, Emre Can Acikgoz, Suvodip Dey, Gokhan Tur, and Dilek Hakkani-Tür. Respact: Harmonizing reasoning, speaking, and acting towards building large language model-based conversational ai agents. arXiv preprint arXiv:2411.00927 , 2024. - · arXiv
  24. [29][29] Hossein Rajaby Faghihi, Aliakbar Nafar, Chen Zheng, Roshanak Mirzaee, Yue Zhang, Andrzej Uszok, Alexander Wan, Tanawan Premsri, Dan Roth, and Parisa Kordjamshidi. Gluecons: A generic benchmark for learning under constraints. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 9552-9561, 2023. -
  25. [30][30] Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Songyang Zhang, Kai Chen, Zongwen Shen, and Jidong Ge. Lawbench: Benchmarking legal knowledge of large language models. arXiv preprint arXiv:2309.16289 , 2023. - · arXiv
  26. [33][33] Angeliki Gioti. Advancements in open source intelligence (osint) techniques and the role of artificial intelligence in cyber threat intelligence (cti). Master's thesis, 2024. -
  27. [34][34] Francesco Greco, Giuseppe Desolda, Andrea Esposito, Alessandro Carelli, et al. David versus goliath: Can machine learning detect llm-generated text? a case study in the detection of phishing emails. In The Italian Conference on CyberSecurity , 2024. -
  28. [35][35] Tudor Groza, Tania Tudorache, Peter N Robinson, and Andreas Zankl. Capturing domain knowledge from multiple sources: the rare bone disorders use case. Journal of Biomedical Semantics , 6:1-15, 2015. -
  29. [36][36] Andreas Happe and Jürgen Cito. Getting pwn'd by ai: Penetration testing with large language models. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages 2082-2086, 2023. -
  30. [37][37] Erik Hemberg, Jonathan Kelly, Michal Shlapentokh-Rothman, Bryn Reinstadler, Katherine Xu, Nick Rutar, and Una-May O'Reilly. Linking threat tactics, techniques, and patterns with defensive weaknesses, vulnerabilities and affected platform configurations for cyber hunting. arXiv preprint arXiv:2010.00533 , 2020. - · arXiv
  31. [38][38] Juan R Bermejo Higuera, Javier Bermejo Higuera, Juan A Sicilia Montalvo, Javier Cubo Villalba, and Juan José Nombela Pérez. Benchmarking approach to compare web applications static analysis tools detecting owasp top ten security vulnerabilities. Computers, Materials & Continua , 64(3), 2020. -
  32. [39][39] Caroline Hillier and Talieh Karroubi. Turning the hunted into the hunter via threat hunting: Life cycle, ecosystem, challenges and the great promise of ai. arXiv preprint arXiv:2204.11076 , 2022. - · arXiv
  33. [40][40] Md Imran Hossen, Jianyi Zhang, Yinzhi Cao, and Xiali Hei. Assessing cybersecurity vulnerabilities in code large language models. arXiv preprint arXiv:2404.18567 , 2024. - · arXiv
  34. [41][41] Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jianguang Lou, Qingwei Lin, Ping Luo, and Saravan Rajmohan. Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation. arXiv preprint arXiv:2408.00764 , 2024. - · arXiv
  35. [43][43] Hyeongyo Jeong, Haechan Lee, Changwon Kim, and Sungtae Shin. A survey of robot intelligence with large language models. Applied Sciences , 14(19):8868, 2024. -
  36. [44][44] Hangyuan Ji, Jian Yang, Linzheng Chai, Chaoren Wei, Liqun Yang, Yunlong Duan, Yunli Wang, Tianzhen Sun, Hongcheng Guo, Tongliang Li, et al. Sevenllm: Benchmarking, eliciting, and enhancing abilities of large language models in cyber threat intelligence. arXiv preprint arXiv:2405.03446 , 2024. - · arXiv
  37. [45][45] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 , 2023. - · arXiv
  38. [46][46] Rabiah Abdul Kadir, Ely Salwana Mat Surin, and Mahidur R Sarker. A systematic review of automated classification for simple and complex query sql on nosql database. Computer Systems Science & Engineering , 48(6), 2024. -
  39. [47][47] Aditya Kulkarni, Vivek Balachandran, Dinil Mon Divakaran, and Tamal Das. From ml to llm: Evaluating the robustness of phishing webpage detection models against adversarial attacks. arXiv preprint arXiv:2407.20361 , 2024. - · arXiv
  40. [49][49] Ziyang Li, Saikat Dutta, and Mayur Naik. Iris: Llm-assisted static analysis for detecting security vulnerabilities. In The Thirteenth International Conference on Learning Representations , 2025. -
  41. [50][50] Xiaoqun Liu, Jiacheng Liang, Qiben Yan, Jiyong Jang, Sicheng Mao, Muchao Ye, Jinyuan Jia, and Zhaohan Xi. Cyber defense reinvented: Large language models as threat intelligence copilots. arXiv preprint arXiv:2502.20791 , 2025. - · arXiv
  42. [51][51] Guilong Lu, Xiaolin Ju, Xiang Chen, Wenlong Pei, and Zhilong Cai. Grace: Empowering llmbased software vulnerability detection with graph structure and in-context learning. Journal of Systems and Software , 212:112031, 2024. -
  43. [52][52] Mary M Lucas, Justin Yang, Jon K Pomeroy, and Christopher C Yang. Reasoning with large language models for medical question answering. Journal of the American Medical Informatics Association , 31(9):1964-1975, 2024. -
  44. [54][54] Sean McGregor, Allyson Ettinger, Nick Judd, Paul Albee, Liwei Jiang, Kavel Rao, William H Smith, Shayne Longpre, Avijit Ghosh, Christopher Fiorelli, et al. To err is ai: A case study informing llm flaw reporting practices. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 28938-28945, 2025. -
  45. [61][61] John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 119-126, 2020. -
  46. [62][62] Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation. arXiv preprint arXiv:2305.15852 , 2023. - · arXiv
  47. [63][63] Lajos Muzsai, David Imolai, and András Lukács. Hacksynth: Llm agent and evaluation framework for autonomous penetration testing. arXiv preprint arXiv:2412.01778 , 2024. - · arXiv
  48. [66][66] Yaroslav Oliinyk, Michael Scott, Ryan Tsang, Chongzhou Fang, Houman Homayoun, et al. Fuzzing { BusyBox } : Leveraging { LLM } and crash reuse for embedded bug unearthing. In 33rd USENIX Security Symposium (USENIX Security 24) , pages 883-900, 2024. -
  49. [69][69] Filippo Perrina, Francesco Marchiori, Mauro Conti, and Nino Vincenzo Verde. Agir: Automating cyber threat intelligence reporting with natural language generation. In 2023 IEEE International Conference on Big Data (BigData) , pages 3053-3062. IEEE, 2023. -
  50. [70][70] Xingzhi Qian, Xinran Zheng, Yiling He, Shuo Yang, and Lorenzo Cavallaro. Lamd: Context-driven android malware detection and classification with llms. arXiv preprint arXiv:2502.13055 , 2025. - · arXiv
  51. [71][71] Jeyavijayan Rajendran, Vinayaka Jyothi, and Ramesh Karri. Blue team red team approach to hardware trust assessment. In 2011 IEEE 29th international conference on computer design (ICCD) , pages 285-288. IEEE, 2011. -
  52. [75][75] Ann Marie Reinhold, Brittany Boles, A Redempta Manzi Muneza, Thomas McElroy, and Clemente Izurieta. Surmounting challenges in aggregating results from static analysis tools. Military Cyber Affairs , 7(1):5-11, 2024. -
  53. [76][76] Rebecca Russell, Louis Kim, Lei Hamilton, Tomo Lazovich, Jacob Harer, Onur Ozdemir, Paul Ellingwood, and Marc McConley. Automated vulnerability detection in source code using deep representation learning. In 2018 17th IEEE international conference on machine learning and applications (ICMLA) , pages 757-762. IEEE, 2018. -
  54. [77][77] Clemens Sauerwein, Christian Sillaber, Andrea Mussmann, and Ruth Breu. A framework for cyber threat hunting. In ACM CCS Workshop on Security and Privacy Analytics , 2019. -
  55. [78][78] Devesh Sawant, Manjesh K Hanawal, and Atul Kabra. Improving discovery of known software vulnerability for enhanced cybersecurity. arXiv preprint arXiv:2412.16607 , 2024. - · arXiv
  56. [81][81] Xiangmin Shen, Lingzhi Wang, Zhenyuan Li, Yan Chen, Wencheng Zhao, Dawei Sun, Jiashui Wang, and Wei Ruan. Pentestagent: Incorporating llm agents to automated penetration testing. arXiv preprint arXiv:2411.05185 , 2024. - · arXiv
  57. [82][82] Ze Sheng, Fenghua Wu, Xiangwu Zuo, Chao Li, Yuxin Qiao, and Lei Hang. Lprotector: An llm-driven vulnerability detection system. arXiv preprint arXiv:2411.06493 , 2024. - · arXiv
  58. [84][84] Adi Simhi, Itay Itzhak, Fazl Barez, Gabriel Stanovsky, and Yonatan Belinkov. Trust me, i'm wrong: High-certainty hallucinations in llms. arXiv preprint arXiv:2502.12964 , 2025. - · arXiv
  59. [85][85] Fahim Sufi. An innovative gpt-based open-source intelligence using historical cyber incident reports. Natural Language Processing Journal , 7:100074, 2024. -
  60. [88][88] Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain, Tamas Bisztray, and Merouane Debbah. Cybermetric: a benchmark dataset based on retrieval-augmented generation for evaluating llms in cybersecurity knowledge. In 2024 IEEE International Conference on Cyber Security and Resilience (CSR) , pages 296-302. IEEE, 2024. -
  61. [89][89] Krist Vaesen and Wybo Houkes. A new framework for teaching scientific reasoning to students from application-oriented sciences. European journal for philosophy of science , 11(2):56, 2021. -
  62. [92][92] Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001 , 2022. - · arXiv
  63. [93][93] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2024. -
  64. [94][94] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824-24837, 2022. -
  65. [95][95] Yue Yang, Fan-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, et al. Holodeck: Language guided generation of 3d embodied ai environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16227-16237, 2024. -
  66. [96][96] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems , 36:11809-11822, 2023. -
  67. [97][97] Ting Yu, Simeon Simoff, and Tony Jan. Vqsvm: a case study for incorporating prior domain knowledge into inductive machine learning. Neurocomputing , 73(13-15):2614-2623, 2010. -
  68. [98][98] Zihan Yu, Liang He, Zhen Wu, Xinyu Dai, and Jiajun Chen. Towards better chain-of-thought prompting strategies: A survey. arXiv preprint arXiv:2310.04959 , 2023. - · arXiv
  69. [99][99] Jie Zhang, Haoyu Bu, Hui Wen, Yongji Liu, Haiqiang Fei, Rongrong Xi, Lun Li, Yun Yang, Hongsong Zhu, and Dan Meng. When llms meet cybersecurity: A systematic literature review. Cybersecurity , 8(1):1-41, 2025. -
  70. [100][100] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations (ICLR) , 2020. Official implementation: https://github.com/Tiiiger/bert\_score . -
  71. [101][101] Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations , 2020. -

Cited by

Sorted by citations

Authors
Liu, X., Yu, F., Li, X., Yan, G., Yang, P., & Xi, Z.
Venue
arXiv
Publisher
arXiv
Published
2025
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2505.11901
arXiv
2505.11901
OpenAlex
W4417300137
Cited by
0
References
0
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
  • Physical Sciences › Computer Science › Information Systems › Information and Cyber Security
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
large language models, LLM benchmarking, embodied AI, function calling, threat attribution, incident response

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review