CoChat
Sign Up
PreprintCited by 56Topic ModelingNatural Language Processing TechniquesLogic, Reasoning, and Knowledge

MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning

Open access
Karpas, Ehud, Abend, Omri, Belinkov, Yonatan, Lenz, Barak, Lieber, Opher, Ratner, Nir, Shoham, Yoav, Bata, Hofit +9 more·2022·arXiv
DOI 10.48550/arxiv.2205.00445arXiv 2205.00445
Open this paper in CoChat — free accountOpen PDF

Abstract

Huge language models (LMs) have ushered in a new era for AI, serving as a gateway to natural-language-based knowledge tasks. Although an essential element of modern AI, LMs are also inherently limited in a number of ways. We discuss these limitations and how they can be avoided by adopting a systems approach. Conceptualizing the challenge as one that involves knowledge and reasoning in addition to linguistic processing, we define a flexible architecture with multiple neural models, complemented by discrete knowledge and reasoning modules. We describe this neuro-symbolic architecture, dubbed the Modular Reasoning, Knowledge and Language (MRKL, pronounced "miracle") system, some of the technical challenges in implementing it, and Jurassic-X, AI21 Labs' MRKL system implementation.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2205.00445

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Understanding when and how to utilize external symbolic tools is important, and this knowledge is dependent on the capabilities of LLMs.

    “The study by MRKL Karpas et al. [90] emphasized the importance of understanding when and how to utilize external symbolic tools, as this knowledge is dependent on the capabilities of LLMs, particularly when these tools can reliably perform functions.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [90]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Research works printed but not linked: 20

  1. [1]1. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (Association for Computational Linguistics, Minneapolis, Minnesota, June 2019), 4171-4186. https://aclanthology.org/N19-1423 .
  2. [2]2. Brown, T. B. et al. Language Models are Few-Shot Learners 2020. https: //arxiv.org/abs/2005.14165 .
  3. [4]4. Chowdhery, A. et al. PaLM: Scaling Language Modeling with Pathways 2022. https://arxiv.org/abs/2204.02311 . · arXiv
  4. [5]5. Raffel, C. et al. Exploring the Limits of Transfer Learning with a Unified Textto-Text Transformer. Journal of Machine Learning Research 21, 1-67. http: //jmlr.org/papers/v21/20-074.html (2020).
  5. [6]6. Sanh, V. et al. Multitask Prompted Training Enables Zero-Shot Task Generalization in International Conference on Learning Representations (2022). https: //openreview.net/forum?id=9Vrb9D0WI4 .
  6. [7]7. Aribandi, V. et al. ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning in International Conference on Learning Representations (2022). https: //openreview.net/forum?id=Vzh1BFUCiIX .
  7. [8]8. Liu, Y. et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach 2019. https://arxiv.org/abs/1907.11692 . · arXiv
  8. [9]9. Smith, S. et al. Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model 2022. https://arxiv.org/ abs/2201.11990 . · arXiv
  9. [10]10. Bommasani, R. et al. On the Opportunities and Risks of Foundation Models 2021. https://arxiv.org/abs/2108.07258 . · arXiv
  10. [11]11. Petroni, F. et al. Language Models as Knowledge Bases? 2019. https://arxiv. org/abs/1909.01066 .
  11. [12]12. Jiang, Z., Xu, F. F., Araki, J. & Neubig, G. How Can We Know What Language Models Know? 2019. https://arxiv.org/abs/1911.12543 . · arXiv
  12. [13]13. Tamkin, A., Brundage, M., Clark, J. & Ganguli, D. Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models 2021. https://arxiv.org/abs/2102.02503 . · arXiv
  13. [14]14. Loureiro, D., Barbieri, F., Neves, L., Anke, L. E. & Camacho-Collados, J. TimeLMs: Diachronic Language Models from Twitter 2022. https://arxiv. org/abs/2202.03829 .
  14. [15]15. Wei, J. et al. Finetuned Language Models Are Zero-Shot Learners 2021. https: //arxiv.org/abs/2109.01652 .
  15. [16]16. Min, S., Lewis, M., Zettlemoyer, L. & Hajishirzi, H. MetaICL: Learning to Learn In Context 2021. https://arxiv.org/abs/2110.15943 . · arXiv
  16. [17]17. Sharir, O., Peleg, B. & Shoham, Y. The Cost of Training NLP Models: A Concise Overview 2020. https://arxiv.org/abs/2004.08900 . · arXiv
  17. [18]18. Wu, C.-J. et al. Sustainable AI: Environmental Implications, Challenges and Opportunities 2021. https://arxiv.org/abs/2111.00364 . · arXiv
  18. [19]19. Strubell, E., Ganesh, A. & McCallum, A. Energy and Policy Considerations for Deep Learning in NLP in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (Association for Computational Linguistics, Florence, Italy, July 2019), 3645-3650. https://aclanthology.org/P191355 .
  19. [20]20. Wolfson, T. et al. Break It Down: A Question Understanding Benchmark 2020. https://arxiv.org/abs/2001.11770 . · arXiv
  20. [21]21. Lester, B., Al-Rfou, R. & Constant, N. The Power of Scale for ParameterEfficient Prompt Tuning 2021. https://arxiv.org/abs/2104.08691 . · arXiv

Cited by

Sorted by citations

Related

Authors
Karpas, E., Abend, O., Belinkov, Y., Lenz, B., Lieber, O., Ratner, N., Shoham, Y., Bata, H., Levine, Y., Leyton-Brown, K., Muhlgay, D., Rozen, N., Schwartz, E., Shachaf, G., Shalev-Shwartz, S., Shashua, A., & Tenenholtz, M.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2205.00445
arXiv
2205.00445
OpenAlex
W4225405251
Cited by
49
References
0
Citations by year
2026: 5 · 2025: 13 · 2024: 18 · 2023: 11 · 2022: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
Keywords
neuro-symbolic architecture, large language models, modular reasoning, external knowledge sources, system architecture

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review