CoChat
Sign Up
PreprintCited by 170Topic ModelingInformation Retrieval and Search BehaviorMultimodal Machine Learning Applications

Knowledge Graph Prompting for Multi-Document Question Answering

Open access
Wang, Yu, Lipka, Nedim, Rossi, Ryan A., Siu, Alexa, Zhang, Ruiyi, Derr, Tyler·2023·arXiv
DOI 10.48550/arxiv.2308.11730arXiv 2308.11730
Open this paper in CoChat — free accountOpen PDF

Abstract

The `pre-train, prompt, predict' paradigm of large language models (LLMs) has achieved remarkable success in open-domain question answering (OD-QA). However, few works explore this paradigm in the scenario of multi-document question answering (MD-QA), a task demanding a thorough understanding of the logical associations among the contents and structures of different documents. To fill this crucial gap, we propose a Knowledge Graph Prompting (KGP) method to formulate the right context in prompting LLMs for MD-QA, which consists of a graph construction module and a graph traversal module. For graph construction, we create a knowledge graph (KG) over multiple documents with nodes symbolizing passages or document structures (e.g., pages/tables), and edges denoting the semantic/lexical similarity between passages or intra-document structural relations. For graph traversal, we design an LLM-based graph traversal agent that navigates across nodes and gathers supporting passages assisting LLMs in MD-QA. The constructed graph serves as the global ruler that regulates the transitional space among passages and reduces retrieval latency. Concurrently, the graph traversal agent acts as a local navigator that gathers pertinent context to progressively approach the question and guarantee retrieval quality. Extensive experiments underscore the efficacy of KGP for MD-QA, signifying the potential of leveraging graphs in enhancing the prompt design for LLMs. Our code: https://github.com/YuWVandy/KG-LLM-MDQA.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2308.11730

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Overly long contexts can lead LLM to the 'Lost in the middle' problem.

    “and overly long contexts can also lead LLM to the”

    From Retrieval-Augmented Generation for Large Language Models: A Survey (2023) · cites this paper as [98]

    The KNN-ST/MDR KG Construction section says that retrieving redundant passages can compromise LLMs’ multi-document question-answering capability. That supports the general concern about excessive context, but our copy does not identify the specific “lost in the middle” problem or say that information in the middle of a long context is overlooked. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Existing agent benchmarks mostly focus on autonomous performance in domains such as web search, coding, search and query, and reasoning.

    “Benchmarks for agents have largely focused on autonomous agent performance in a variety of domains such asweb search (38), coding (22), search and query (51), and reasoning (59; 33).”

    From Project Sid: Many-agent simulations toward AI civilization (2024) · cites this paper as [51]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 43

  1. Aly, R.; Guo, Z.; Schlichtkrull, M.; Thorne, J.; Vlachos, A.; Christodoulopoulos, C.; Cocarascu, O.; and Mittal, A. 2021. FEVEROUS: Fact extraction and VERification over unstructured and structured information. arXiv preprint arXiv:2106.05707. · arXiv
  2. Asai, A.; Hashimoto, K.; Hajishirzi, H.; Socher, R.; and Xiong, C. 2019. Learning to retrieve reasoning paths over wikipedia graph for question answering. arXiv preprint arXiv:1911.10470. · arXiv
  3. Bolino, M.; Long, D.; and Turnley, W. 2016. Impression management in organizations: Critical questions, answers, and areas for future research. Annual Review of Organi- zational Psychology and Organizational Behavior, 3: 377- 406.
  4. Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Ad- vances in neural information processing systems, 33: 1877- 1901.
  5. Caciularu, A.; Peters, M. E.; Goldberger, J.; Dagan, I.; and Cohan, A. 2023. Peek Across: Improving Multi-Document Modeling via Cross-Document Question-Answering. arXiv preprint arXiv:2305.15387. · arXiv
  6. Chen, D.; Fisch, A.; Weston, J.; and Bordes, A. 2017. Read- ing wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051. · arXiv
  7. Chung, H. W.; Hou, L.; Longpre, S.; Zoph, B.; Tay, Y.; Fe- dus, W.; Li, E.; Wang, X.; Dehghani, M.; Brahma, S.; et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. · arXiv
  8. Gururangan, S.; Marasović, A.; Swayamdipta, S.; Lo, K.; Beltagy, I.; Downey, D.; and Smith, N. A. 2020. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964. · arXiv
  9. Ho, X.; Nguyen, A.-K. D.; Sugawara, S.; and Aizawa, A. 2020. Constructing a multi-hop QA dataset for com- prehensive evaluation of reasoning steps. arXiv preprint arXiv:2011.01060. · arXiv
  10. Ji, Z.; Lee, N.; Frieske, R.; Yu, T.; Su, D.; Xu, Y.; Ishii, E.; Bang, Y. J.; Madotto, A.; and Fung, P. 2023. Survey of hal- lucination in natural language generation. ACM Computing Surveys, 55(12): 1-38.
  11. Jiang, Y.; and Bansal, M. 2019. Avoiding reasoning short- cuts: Adversarial evaluation, training, and model develop- ment for multi-hop QA. arXiv preprint arXiv:1906.07132. · arXiv
  12. Jin, B.; Liu, G.; Han, C.; Jiang, M.; Ji, H.; and Han, J. 2023. Large Language Models on Graphs: A Comprehensive Sur- vey. arXiv preprint arXiv:2312.02783. · arXiv
  13. Ju, M.; Yu, W.; Zhao, T.; Zhang, C.; and Ye, Y. 2022. Grape: Knowledge graph enhanced passage reader for open-domain question answering. arXiv preprint arXiv:2210.02933. · arXiv
  14. Karpukhin, V.; Oguz, B.; Min, S.; Lewis, P.; Wu, L.; Edunov, S.; Chen, D.; and Yih, W.-t. 2020. Dense passage re- trieval for open-domain question answering. arXiv preprint arXiv:2004.04906. · arXiv
  15. Liu, P.; Yuan, W.; Fu, J.; Jiang, Z.; Hayashi, H.; and Neubig, G. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9): 1-35.
  16. Liu, X.; Dong, Z.; and Zhang, P. 2023. Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering. arXiv preprint arXiv:2310.06238. · arXiv
  17. Mao, Y.; He, P.; Liu, X.; Shen, Y.; Gao, J.; Han, J.; and Chen, W. 2021. RIDER: Reader-Guided Passage Rerank- ing for Open-Domain Question Answering. arXiv preprint arXiv:2101.00294. · arXiv
  18. Mathew, M.; Karatzas, D.; and Jawahar, C. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2200-2209.
  19. Min, S.; Chen, D.; Zettlemoyer, L.; and Hajishirzi, H. 2019. Knowledge guided text retrieval and reading for open do- main question answering. arXiv preprint arXiv:1911.03868. · arXiv
  20. Pandya, H. A.; and Bhatt, B. S. 2021. Question answering survey: Directions, challenges, datasets, evaluation matrices. arXiv preprint arXiv:2112.03572. · arXiv
  21. Pereira, J.; Fidalgo, R.; Lotufo, R.; and Nogueira, R. 2023. Visconde: Multi-document QA with GPT-3 and Neural Reranking. In European Conference on Information Re- trieval, 534-543. Springer.
  22. Qin, C.; Zhang, A.; Zhang, Z.; Chen, J.; Yasunaga, M.; and Yang, D. 2023. Is ChatGPT a general-purpose nat- ural language processing task solver? arXiv preprint arXiv:2302.06476. · arXiv
  23. Ramos, J.; et al. 2003. Using tf-idf to determine word rele- vance in document queries. In Proceedings of the first in- structional conference on machine learning, volume 242, 29-48.
  24. Robertson, S.; Zaragoza, H.; et al. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval, 3(4): 333-389.
  25. Tessuto, G. 2011. Legal problem question answer genre across jurisdictions and cultures. English for Specific Pur- poses, 30(4): 298-309.
  26. Thorne, J.; Vlachos, A.; Christodoulopoulos, C.; and Mittal, A. 2018. FEVER: a large-scale dataset for fact extraction and VERification. arXiv preprint arXiv:1803.05355. · arXiv
  27. Tian, Y.; Dong, K.; Zhang, C.; Zhang, C.; and Chawla, N. V. 2023a. Heterogeneous graph masked autoencoders. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 37, 9997-10005.
  28. Tian, Y.; Song, H.; Wang, Z.; Wang, H.; Hu, Z.; Wang, F.; Chawla, N. V.; and Xu, P. 2023b. Graph neural prompting with large language models. arXiv preprint arXiv:2309.15427. · arXiv
  29. Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; et al. 2023. Llama: Open and efficient founda- tion language models. arXiv preprint arXiv:2302.13971. · arXiv
  30. Trivedi, H.; Balasubramanian, N.; Khot, T.; and Sabharwal, A. 2022a. Interleaving retrieval with chain-of-thought rea- soning for knowledge-intensive multi-step questions. arXiv preprint arXiv:2212.10509. · arXiv
  31. Trivedi, H.; Balasubramanian, N.; Khot, T.; and Sabharwal, A. 2022b. MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics, 10: 539-554.
  32. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of- thought prompting elicits reasoning in large language mod- els. Advances in Neural Information Processing Systems, 35: 24824-24837.
  33. Wu, X.; Zhou, K.; Sun, M.; Wang, X.; and Liu, N. 2023. A survey of graph prompting methods: techniques, applica- tions, and challenges. arXiv preprint arXiv:2303.07275. · arXiv
  34. Xiong, W.; Li, X. L.; Iyer, S.; Du, J.; Lewis, P.; Wang, W. Y.; Mehdad, Y.; Yih, W.-t.; Riedel, S.; Kiela, D.; et al. 2020. Answering complex open-domain questions with multi-hop dense retrieval. arXiv preprint arXiv:2009.12756. · arXiv
  35. Xu, Y.; Xu, Y.; Lv, T.; Cui, L.; Wei, F.; Wang, G.; Lu, Y.; Florencio, D.; Zhang, C.; Che, W.; et al. 2020. Layoutlmv2: Multi-modal pre-training for visually-rich document under- standing. arXiv preprint arXiv:2012.14740. · arXiv
  36. Yang, J.; Jin, H.; Tang, R.; Han, X.; Feng, Q.; Jiang, H.; Yin, B.; and Hu, X. 2023. Harnessing the power of llms in practice: A survey on chatgpt and beyond. arXiv preprint arXiv:2304.13712. · arXiv
  37. Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T. L.; Cao, Y.; and Narasimhan, K. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601. · arXiv
  38. Yao, Y.; Li, Z.; and Zhao, H. 2023. Beyond Chain-of- Thought, Effective Graph-of-Thought Reasoning in Large Language Models. arXiv preprint arXiv:2305.16582. · arXiv
  39. Yavuz, S.; Hashimoto, K.; Zhou, Y.; Keskar, N. S.; and Xiong, C. 2022. Modeling multi-hop question an- swering as single sequence prediction. arXiv preprint arXiv:2205.09226. · arXiv
  40. Yu, Y.; Zhuang, Y.; Zhang, J.; Meng, Y.; Ratner, A.; Krishna, R.; Shen, J.; and Zhang, C. 2023. Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias. arXiv preprint arXiv:2306.15895. · arXiv
  41. Yuan, Z.; Yuan, H.; Tan, C.; Wang, W.; and Huang, S. 2023. How well do Large Language Models perform in Arithmetic tasks? arXiv preprint arXiv:2304.02015. · arXiv
  42. Zhu, F.; Lei, W.; Wang, C.; Zheng, J.; Poria, S.; and Chua, T.-S. 2021. Retrieving and reading: A comprehensive sur- vey on open-domain question answering. arXiv preprint arXiv:2101.00774. · arXiv
  43. Zou, H.; and Caragea, C. 2023. JointMatch: A Unified Approach for Diverse and Collaborative Pseudo-Labeling to Semi-Supervised Text Classification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, 7290-7301.

Cited by

Sorted by citations

Authors
Wang, Y., Lipka, N., Rossi, R. A., Siu, A., Zhang, R., & Derr, T.
Affiliations
  • Vanderbilt University (US)
  • Adobe Systems (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Volume
38
Issue
17
Pages
19206-19214
ISSN
2159-5399 (ISSN-L)
DOI
10.48550/arxiv.2308.11730
arXiv
2308.11730
OpenAlex
W4393152682
Cited by
170
References
0
Citations by year
2026: 64 · 2025: 75 · 2024: 30 · 2023: 1
FWCI
49.80
Type
conference paper
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • Proceedings of the AAAI Conference on Artificial Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (2239881)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
multi-document question answering, large language models, graph construction, passage retrieval, semantic similarity, retrieval-augmented generation, prompt engineering

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review