CoChat
Sign Up
PreprintCited by 12Topic ModelingInformation Retrieval and Search BehaviorSpeech and dialogue systems

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

FACTS About Building Retrieval Augmented Generation-based Chatbots

Open access
Akkiraju, Rama, Xu, Anbang, Bora, Deepak, Yu, Tan, An, Lu, Seth, Vishal, Shukla, Aaditya, Gundecha, Pritam, Mehta, Hridhay, Jha, Ashwin, Raj, Prithvi, Balasubramanian, Abhinav, Maram, Murali, Muthusamy, Guru, Annepally, Shivakesh Reddy, Knowles, Sidney, Du, Min, Burnett, Nick, Javiya, Sean, Marannan, Ashok, Kumari, Mamta, Jha, Surbhi, Dereszenski, Ethan, Chakraborty, Anupam, Ranjan, Subhash, Terfai, Amina, Surya, Anoop, Mercer, Tracey, Thanigachalam, Vinodh Kumar, Bar, Tamar, Krishnan, Sanjana, Kilaru, Samy, Jaksic, Jasmine, Algarici, Nave, Liberman, Jacob, Conway, Joey, Nayyar, Sonu, Boitano, Justin·2024·arXiv
DOI 10.48550/arxiv.2407.07858arXiv 2407.07858
Open this paper in CoChat — free accountOpen PDF

Abstract

Enterprise chatbots, powered by generative AI, are emerging as key applications to enhance employee productivity. Retrieval Augmented Generation (RAG), Large Language Models (LLMs), and orchestration frameworks like Langchain and Llamaindex are crucial for building these chatbots. However, creating effective enterprise chatbots is challenging and requires meticulous RAG pipeline engineering. This includes fine-tuning embeddings and LLMs, extracting documents from vector databases, rephrasing queries, reranking results, designing prompts, honoring document access controls, providing concise responses, including references, safeguarding personal information, and building orchestration agents. We present a framework for building RAG-based chatbots based on our experience with three NVIDIA chatbots: for IT/HR benefits, financial earnings, and general content. Our contributions are three-fold: introducing the FACTS framework (Freshness, Architectures, Cost, Testing, Security), presenting fifteen RAG pipeline control points, and providing empirical results on accuracy-latency tradeoffs between large and small LLMs. To the best of our knowledge, this is the first paper of its kind that provides a holistic view of the factors as well as solutions for building secure enterprise-grade chatbots."

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2407.07858

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Memory types in Agentic AI include episodic memory, semantic memory, and vector-based memory for RAG.

    “Memory types include episodic memory (task-specific history) [167,168], semantic memory (long-term facts or structured data) [169,170], and vector-based memory for RAG [171, 172].”

    From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [172]

    The Introduction (page unavailable) identifies vector-search retrieval as a core component of RAG and mentions memory among chatbot orchestration features. It does not identify vector-based retrieval as a type of agentic memory; the episodic and semantic memory clauses are attached to other citations. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 13

  1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). · arXiv
  2. Barnett, S., Kurniawan, S., Thudumu, S., Brannelly, Z., and Abdelrazek, M. Seven failure points when engineering a retrieval augmented generation system. arXiv preprint arXiv:2401.05856 (2024). · arXiv
  3. Es, S., James, J., Espinosa-Anke, L., and Schockaert, S. Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217 (2023). · arXiv
  4. Galitsky, B. Developing enterprise chatbots. Springer, 2019.
  5. Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., and Neubig, G. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983 (2023). · arXiv
  6. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küt- tler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 (2020), 9459-9474.
  7. Liu, M., Ene, T.-D., Kirby, R., Cheng, C., Pinckney, N., Liang, R., Alben, J., Anand, H., Banerjee, S., Bayraktaroglu, I., et al. Chipnemo: Domain-adapted llms for chip design. arXiv preprint arXiv:2311.00176 (2023). · arXiv
  8. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems 36 (2024).
  9. Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842 (2023). · arXiv
  10. Rebedea, T., Dinu, R., Sreedhar, M., Parisien, C., and Cohen, J. Nemo guardrails: A toolkit for controllable and safe llm applications with programmable rails. arXiv preprint arXiv:2310.10501 (2023). · arXiv
  11. Saad-Falcon, J., Khattab, O., Potts, C., and Zaharia, M. Ares: An automated evaluation framework for retrieval-augmented generation systems. arXiv preprint arXiv:2311.09476 (2023). · arXiv
  12. Setty, S., Jijo, K., Chung, E., and Vidra, N. Improving retrieval for rag based question answering models on financial documents. arXiv preprint arXiv:2404.07221 (2024). · arXiv
  13. Zhu, Y., Yuan, H., Wang, S., Liu, J., Liu, W., Deng, C., Dou, Z., and Wen, J.- R. Large language models for information retrieval: A survey. arXiv preprint arXiv:2308.07107 (2023). · arXiv

Cited by

Sorted by citations

Metadata

Authors
Akkiraju, R., Xu, A., Bora, D., Yu, T., An, L., Seth, V., Shukla, A., Gundecha, P., Mehta, H., Jha, A., Raj, P., Balasubramanian, A., Maram, M., Muthusamy, G., Annepally, S. R., Knowles, S., Du, M., Burnett, N., Javiya, S., Marannan, A., Kumari, M., Jha, S., Dereszenski, E., Chakraborty, A., Ranjan, S., Terfai, A., Surya, A., Mercer, T., Thanigachalam, V. K., Bar, T., Krishnan, S., Kilaru, S., Jaksic, J., Algarici, N., Liberman, J., Conway, J., Nayyar, S., & Boitano, J.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2407.07858
arXiv
2407.07858
OpenAlex
W4400601280
Cited by
12
References
1
Citations by year
2026: 2 · 2025: 9 · 2024: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
retrieval-augmented generation, large language models, embedding fine-tuning, document access control, prompt engineering, personal data protection, accuracy-latency trade-off, chatbot security

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review