The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
Measuring and Narrowing the Compositionality Gap in Language Models
Open accessAbstract
We investigate the ability of language models to perform compositional reasoning tasks where the overall solution depends on correctly composing the answers to sub-problems.We measure how often models can correctly answer all sub-problems but not generate the overall solution, a ratio we call the compositionality gap.We evaluate this ratio by asking multi-hop questions with answers that require composing multiple facts unlikely to have been observed together during pretraining.In the GPT-3 family of models, as model size increases we show that the single-hop question answering performance improves faster than the multi-hop performance does, therefore the compositionality gap does not decrease.This surprising result suggests that while more powerful models memorize and recall more factual knowledge, they show no corresponding improvement in their ability to perform this kind of compositional reasoning.We then demonstrate how elicitive prompting (such as chain of thought) narrows the compositionality gap by reasoning explicitly.We present a new method, self-ask, that further improves on chain of thought.In our method, the model explicitly asks itself (and answers) follow-up questions before answering the initial question.We finally show that self-ask's structured prompting lets us easily plug in a search engine to answer the follow-up questions, which additionally improves accuracy.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Systematicity continues to challenge models11–18 and motivates new frameworks34–41.”
From Human-like systematic generalization through a meta-learning neural network (2023) · cites this paper as [18]
The cited source says what the claim attributes to it.
In the citing paper
“Notably, modern neural networks still struggle on tests of systematicity11–18—tests that even a minimally algebraic mind should pass2.”
From Human-like systematic generalization through a meta-learning neural network (2023) · cites this paper as [18]
In the citing paper
“recent studies have demonstrated the limited generalization capabilities of LLMs, with deficiencies in planning15, abstraction16, and compositionality17 across various tasks.”
From Limitations of large language models in clinical problem-solving arising from inflexible reasoning (2025) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Self-citationA Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1014 · Cited by 2,459
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Vinyals, O., & Le, Q. (2015) · arXiv · doi:10.48550/arxiv.1506.05869 · Cited by 1,400
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Personalizing Dialogue Agents: I have a dog, do you have pets too?
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., & Weston, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1205 · Cited by 1,177
Neural Responding Machine for Short-Text Conversation
Shang, L., Lu, Z., & Li, H. (2015) · doi:10.3115/v1/p15-1152 · Cited by 1,146
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.74 · Cited by 1,109
Decomposed Prompting: A Modular Approach for Solving Complex TasksPreprint
Khot, T., Trivedi, H., Finlayson, M., Fu, Y., Richardson, K., Clark, P., & Sabharwal, A. (2022) · International Conference on Learning Representations · doi:10.48550/arxiv.2210.02406 · Cited by 852
Recipes for Building an Open-Domain Chatbot
Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., et al. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.24 · Cited by 784
A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J., Gao, J., & Dolan, B. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1020 · Cited by 782
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778
Multi-Scale Dense Networks for Resource Efficient Image ClassificationPreprint
Huang, G. M., Chen, D., Li, T., Wu, F., van der Maaten, L. J. P., & Weinberger, K. Q. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.09844 · Cited by 741
SkipNet: Learning Dynamic Routing in Convolutional Networks
Wang, X., Yu, F., Dou, Z.-Y., Darrell, T., & Gonzalez, J. E. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01261-8_25 · Cited by 681
How Much Knowledge Can You Pack into the Parameters of a Language Model?
Roberts, A., Raffel, C., & Shazeer, N. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.437 · Cited by 638
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614
ReAct: Synergizing Reasoning and Acting in Language Models
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.03629 · Cited by 608
REALM: Retrieval-Augmented Language Model Pre-Training
Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020) · arXiv · doi:10.48550/arxiv.2002.08909 · Cited by 574
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
The Web as a Knowledge-Base for Answering Complex Questions
Talmor, A., & Berant, J. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1059 · Cited by 520
The Second Conversational Intelligence Challenge (ConvAI2)
Dinan, E., Logacheva, V., Malykh, V., Miller, A., Shuster, K., Urbanek, J., Kiela, D., Szlam, A. D., Serban, I. V., Lowe, R., et al. (2019) · The Springer series on challenges in machine learning · doi:10.1007/978-3-030-29135-8_7 · Cited by 395
Improving language models by retrieving from trillions of tokens
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Driessche, G. V. D., Lespiau, J.-B., Damoc, B., Clark, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04426 · Cited by 382
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
Zhou, D., Schärli, N., Hou, L., Wei, J., Scales, N., Wang, X., Schuurmans, D., Cui, C., Bousquet, O., Le, Q., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.10625 · Cited by 380
The Second Conversational Intelligence Challenge (ConvAI2)
Dinan, E., Logacheva, V., Malykh, V., Miller, A., Shuster, K., Urbanek, J., Kiela, D., Szlam, A., Serban, I., Lowe, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1902.00098 · Cited by 355
Adaptive Computation Time for Recurrent Neural Networks
Graves, A. (2016) · arXiv · doi:10.48550/arxiv.1603.08983 · Cited by 335
Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems
Ling, W., Yogatama, D., Dyer, C., & Blunsom, P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1015 · Cited by 325
oLMpics-On What Language Model Pre-training Captures
Talmor, A., Elazar, Y., Goldberg, Y., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00342 · Cited by 288
Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
Ho, X., Duong Nguyen, A.-K., Sugawara, S., & Aizawa, A. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.580 · Cited by 287
Generalization through Memorization: Nearest Neighbor Language Models
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00172 · Cited by 285
Self-citation♫ MuSiQue: Multihop Questions via Single-hop Question Composition
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00475 · Cited by 283
WebGPT: Browser-assisted question-answering with human feedback
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.09332 · Cited by 272
Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
Keysers, D., Schärli, N., Scales, N., Buisman, H., Furrer, D., Kashubin, S., Momchev, N., Sinopalnikov, D., Stafiniak, L., Tihon, T., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.09713 · Cited by 264
Adaptive Neural Networks for Efficient Inference
Bolukbasi, T., Wang, J., Dekel, O., & Saligrama, V. (2017) · arXiv · doi:10.48550/arxiv.1702.07811 · Cited by 251
Generated Knowledge Prompting for Commonsense Reasoning
Liu, J., Liu, A., Lu, X., Welleck, S., West, P., Bras, R. J. L., Choi, Y., & Hajishirzi, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.225 · Cited by 229
Unsupervised Commonsense Question Answering with Self-Talk
Shwartz, V., West, P., Bras, R. J. L., Bhagavatula, C., & Choi, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.373 · Cited by 215
Search-based Neural Structured Learning for Sequential Question Answering
Iyyer, M., Yih, W.-T., & Chang, M.-W. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1167 · Cited by 213
Neural Responding Machine for Short-Text ConversationPreprint
Shang, L., Lu, Z., & Li, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.02364 · Cited by 212
Multi-hop Reading Comprehension through Question Decomposition and Rescoring
Min, S., Zhong, V., Zettlemoyer, L., & Hajishirzi, H. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1613 · Cited by 197
Self-citationCompositionality Decomposed: How do Neural Networks Generalise?
Hupkes, D., Dankers, V., Mul, M., & Bruni, E. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.11674 · Cited by 194
The Right Tool for the Job: Matching Model and Instance Complexities
Schwartz, R., Stanovsky, G., Swayamdipta, S., Dodge, J., & Smith, N. A. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.593 · Cited by 163
Self-citationCan You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent NetworksPreprint
Schwarzschild, A., Borgnia, E., Gupta, A., Huang, F., Vishkin, U., Goldblum, M., & Goldstein, T. (2021) · Neural Information Processing Systems · doi:10.48550/arxiv.2106.04537 · Cited by 151
Show Your Work: Scratchpads for Intermediate Computation with Language Models
Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.00114 · Cited by 144
Reframing Instructional Prompts to GPTk’s Language
Mishra, S. R., Khashabi, D., Baral, C. R., Choi, Y., & Hajishirzi, H. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.50 · Cited by 140
Ask the Right Questions: Active Question Reformulation with Reinforcement LearningPreprint
Buck, C., Bulian, J., Ciaramita, M., Gajewski, W., Gesmundo, A., Houlsby, N., & Wang, W. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07830 · Cited by 127
Unsupervised Question Decomposition for Question Answering
Perez, E., Lewis, P., Yih, W.-T., Cho, K., & Kiela, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.713 · Cited by 123
Neural Logic MachinesPreprint
Dong, H., Mao, J., Lin, T., Wang, C., Li, L. C., & Zhou, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.11694 · Cited by 123
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image RecognitionPreprint
Wang, Y., Huang, R., Song, S., Huang, Z., & Huang, G. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.15075 · Cited by 110
Answering Complex Open-domain Questions Through Iterative Query Generation
Qi, P., Lin, X., Mehr, L., Wang, Z., & Manning, C. D. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1261 · Cited by 106
Lake, B., & Baroni, M. (2017) · International Conference on Learning Representations · doi:10.48550/arxiv.1711.00350 · Cited by 85
Knowledge-Aware Language Model PretrainingPreprint
Rosset, C., Xiong, C., Phan, M., Song, X., Bennett, P. N., & Tiwary, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00655 · Cited by 84
Generative Deep Neural Networks for Dialogue: A Short ReviewPreprint
Serban, I. V., Lowe, R., Charlin, L., & Pineau, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.06216 · Cited by 82
Neural Speed Reading via Skim-RNNPreprint
Seo, M., Min, S., Farhadi, A., & Hajishirzi, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.02085 · Cited by 77
Self-citationIs a Question Decomposition Unit All We Need?Preprint
Patel, P. H., Mishra, S., Parmar, M., & Baral, C. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2205.12538 · Cited by 67
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models
Khot, T., Khashabi, D., Richardson, K., Clark, P. E., & Sabharwal, A. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.99 · Cited by 56
Teaching language models to support answers with verified quotes
Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.11147 · Cited by 53
RealTime QA: What's the Answer Right Now?
Kasai, J., Sakaguchi, K., Takahashi, Y., Bras, R. L., Asai, A., Yu, X., Radev, D., Smith, N. A., Choi, Y., & Inui, K. (2022) · arXiv · doi:10.48550/arxiv.2207.13332 · Cited by 52
Self-citationRealTime QA: What's the Answer Right Now?
Kasai, J., Sakaguchi, K., Takahashi, Y., Bras, R. L., Asai, A., Yu, X., Radev, D., Smith, N., Choi, Y., & Inui, K. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2130 · Cited by 52
Self-citationSkipNet: Learning Dynamic Routing in Convolutional NetworksPreprint
Wang, X., Yu, F., Dou, Z.-Y., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.09485 · Cited by 46
Answer-based Adversarial Training for Generating Clarification Questions
Rao, S., & Daumé, H. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1013 · Cited by 40
Break It Down: A Question Understanding BenchmarkPreprint
Wolfson, T., Geva, M., Gupta, A., Gardner, M., Goldberg, Y., Deutch, D., & Berant, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.11770 · Cited by 40
Is a Question Decomposition Unit All We Need?
Patel, P., Mishra, S. R., Parmar, M., & Baral, C. R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.302 · Cited by 37
The Unreliability of Explanations in Few-Shot In-Context LearningPreprint
Ye, X., & Durrett, G. (2022) · arXiv.org · doi:10.48550/arxiv.2205.03401 · Cited by 37
Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2Preprint
Betz, G., Richardson, K., & Voigt, C. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.13033 · Cited by 32
Multi-hop Reading Comprehension through Question Decomposition and RescoringPreprint
Min, S., Zhong, V., Zettlemoyer, L., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02916 · Cited by 14
Self-citationJi, Y., & Gao, S. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 10
Cited by
Sorted by citations
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 2,314
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.03714 · Cited by 911
Augmenting large language models with chemistry tools
M. Bran, A., Cox, S., Schilter, O., Baldassari, C., White, A. D., & Schwaller, P. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00832-8 · Cited by 794
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 561
Metadata
- Authors
- Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., & Lewis, M.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- Corresponding author
- Press, Ofir
- ORCID
- Smith, Noah
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Published
- 2023
- Pages
- 5687-5711
- OpenAlex
- W4389520103
- Cited by
- 303
- References
- 61
- Citations by year
- 2026: 97 · 2025: 90 · 2024: 46 · 2023: 69
- FWCI
- 53.42
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2113530)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- compositional reasoning, multi-hop question answering, chain-of-thought prompting, GPT-3, model size scaling, structured prompting, search engine integration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
